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ABSTRACT 


Marine Corps Manpower Plans and Policy Division, Manpower & Reserve 
Affairs, is responsible for formulating Marine Corps force manpower plans. 
Accomplishing this mission requires extensive knowledge of the Human 
Resource Development Process (HRDP) for controlling future personnel attrition, 
retention, and accession quantities to ensure appropriate quantities of its various 
Military Occupational Specialties (MOS) and overall end strength are maintained. 
To assist their mission, an agent-based computer simulation model was 
developed in the Java computer language. This thesis investigates that 
simulation model, titled Manpower Simulation Model (MSM). This thesis provides 
documentation of MSM’s architecture and processes, tests the sensitivity of its 
inputs through the use of an experimental design, and validates MSM’s output 
measures by calculating the relative error for five successive forecast years for 
various HRDP categories. This thesis found that MSM’s structure and output 
measurement responses aligned with HRDP practices. With respect to validation, 
on average the HRDP categories losses and accessions underestimated by 17 
and 18 percent, respectively, while gains overestimated by 36 percent. The 
category promotions generally underestimated, but lessened in magnitude as 
grade increased. The category retention consistently overestimated for all 
grades. Lastly, the MSM showed biasness toward retaining Marines over 
backfilling vacancies through accessions. 
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I. INTRODUCTION 


A. BACKGROUND 

The Marine Corps is regarded as being the Nation’s premier expeditionary 
force in readiness. Before it can ever be deployed to conduct its various ranges 
of military operations, it must ensure it has the proper personnel in place to 
execute those missions. Annotated as the Human Resources Development 
Process (HRDP), the Marines and civilians at Manpower and Reserve Affairs 
(M&RA) wrestle with this convoluted and mission critical process to ensure the 
Marine Corps has the proper number of Marines proportionately allocated across 
its various Military Occupations Specialties (MOS). To add an additional layer of 
difficultly, M&RA must accomplish this mission for an organization that has an 
extreme flux in personnel change. 

Averaging from the past decade, each year there are approximately 
30,000 Marines that choose to enter and exit the Marine Corps (Spafford, 2016). 
Roughly 60% of Marines are on their first enlistment contract, 45% are below the 
paygrade of E-4, and 75% will not reenlist for a second contract (Spafford, 2016). 
Figure 1 shows this dramatic manpower drop of Enlisted Marines exiting the 
Marine Corps after their first enlistment contract. This significant drop is due to 
the general manpower structure of the Marine Corps, as well as its merit-based 
promotion selection that is in conjunction with the “up or out” manpower system. 
To ensure mission accomplishment within M&RA, it is crucial to understand 
these factors that drive manpower movements. The manpower process is 
expanded on in Section C or this chapter. 
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Year 


Figure 1. Marine Corps Enlisted Force Fiscal Year (FY) 1990-2000. 

Source: Rostker (2013). 

Advances in technology have facilitated the ability to develop tools that 
yield enhanced manpower analysis. Flowever, most of these decision making 
tools are still relatively isolated models that provide insightful information in a 
narrow spectrum of the HRDP (R. A. Garrick, personal communication, April 6, 
2016). Naturally, policy and procedures implemented within M&RA try to mitigate 
the unintended consequences and errors of assembling the various patches of 
information. To bridge this gap, the power of computer simulation was explored 
with the intent to integrate the numerous processes of HRDP into a single 
simulation model. 

The first documented computer modeling software that attempted to 
provide a HRDP whole-system oriented viewpoint was an Arena model. Created 
by Rockwell Automation, Arena is a simulation software that models business 
practices to assist that business in its “ability to analyze and make decisions on 
how to improve (their) process” (Rockwell Automation, 2016). Used at M&RA 

from roughly 2008 to 2011, this model was cumbersome, slow, and complex. 
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Ultimately, its inability to forecast numerous years into the future and 
undocumented source code methods led this software program to become 
untrusted, impractical, and unused to the point that its contracting license with 
Rockwell Automation was not renewed upon its licensing expiration (M. Ramirez, 
personal communication, April 4, 2016). 

In 2011, former Naval Postgraduate School Operations Research 
graduate Captain Ronald Garrick, USMC (ret), arrived at Manpower Studies and 
Analysis Branch, M&RA, thereupon noticing the technology capability gap and 
began developing what ultimately became the Manpower Simulation Model 
(MSM). 

The MSM is a Java language agent-based simulation program that aims 
to integrate the numerous processes of the HRDP under a single computer 
simulation model. Using the Marine Corps’ force structure as its virtual 
architecture and manpower shaping rules as its internal constraints, the program 
combines the deterministic aspects of the HRDP with a stochastic simulation 
capability that produces a multitude of output metrics. These output metrics aid 
the decision makers at M&RA to gain insightful forecasted metrics that range 
across numerous fiscal years, and breaks down into the dimensional analysis at 
the MOS skill and grade level. 

B. PURPOSE 

This thesis provides an overall evaluation of the MSM and its potential for 
future usage so that its output metrics can be used with trust and confidence in 
making manpower decisions by the leaders within M&RA. This was 
accomplished by inspecting and documenting the various internal structures and 
procedures within the MSM. A software-verification approach was taken to test 
the sensitivity of its input metrics through the use of an experimental design. 
Finally, a software-validation approach was used to evaluate the model’s 
accuracy against historical Marine Corps manpower data. 
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C. THE MANPOWER PROCESS 

To understand the underlying concepts of the MSM, and its goal of 
providing forecasted metrics, one first has to understand the enlisted HRDP. The 
intention of this section is to provide the reader with an overview of this process, 
rather than an in-depth detailed analysis, to serve as the basis for comparison to 
the MSM. This section will highlight the conjoined process for Officer and 
Enlisted personnel where appropriate. However, the scope of this thesis is purely 
focused on enlisted personnel, since that is the personnel parameter used within 
the MSM. 

Forecasting manpower quantities are derived from two critical 
components; the absolute manpower structure of the Marine Corps and the 
personnel movement that occurs within it. The first part of this section will focus 
on manpower structure. Comprising of approximately 240 MOSs that are spread 
throughout 9 Enlisted grades, the Marine Corps Enlisted Force averages around 
162,000 Marines at any given time (Manpower Plans and Policy Division, 2016). 

Determining what the force structure should be for a given fiscal year is a 
cumbersome process. The enlisted force is derived from manpower requirements 
or previous force structures. These requirements are determined from 
occupational field (occField) sponsors who are specialized and advocate for 
occFields to possess specific levels of experience, where experience can be 
synonymous with a Marine’s time in service, that are proportionality distributed 
across the Marine Corps in terms of both depth and breadth. The sponsor 
receives guidance from the Commandant of the Marine Corps in his strategic 
analysis of various documents such as the National Security Strategy and the 
National Defense Strategy, as well as from current Combatant Commanders who 
yield bottom-up refinement to previously established force structures (United 
States Marine Corps, 2009). The Deputy Commandant Combat Development 
and Integration (DC CD&I) ultimately approves the new requirements list, which 
appropriately becomes the documented Table of Organization (T/0). Figure 2 
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depicts this process, as described in Marine Corps Order (MCO) 5311.1D Total 
Force Structure Process. 


Input 


Analysis 


Output 



Tasks 

Conditions 

Standards 


Current 


GAPS 


Table of Organization 
and Equipment 
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Manpower 


Equipment 


Training 



[jFSM^ 


Figure 2. The Marine Corps Total Force Structure Process. Adapted from 

United States Marine Corps (2009). 


The T/0 is therefore the foundational document toward which the Marine 
Corps tries to build and sustain its force, which is accomplished by “buying billet 
seats.” Every purchased billet has political and budgetary constraints associated 
with it. The process of “buying billet seats” is synonymous with the fiscal year’s 
approved end strength as provided by the National Defense Authorization Act 
(U.S. Department of Defense, 1996). When differences between what the Marine 
Corps would like to have, the T/0, and what the Marine Corps is allowed to have, 
its approved end strength, the Manning and Staffing Precedence Order is taken 
into account. The Manning and Staffing Precedence Order is a MCO that 
identifies which units will receive priority in fulfilling billet vacancies, as directed 
by the Commandant of the Marine Corps (United States Marine Corps, 2012). 
The combination of these steps in addition to the original T/0 produces the 
Authorized Strength Report (ASR). 

The ASR is the principal document that M&RA receives to complete the 
final components of the FIRDP. However, before it can be used it has to be 
reconciled to be compliant with political and budgetary constraints. For budgetary 
constraints, the grade quantities are aggregated and summed with respect to that 

grade’s pay to ensure that monetary personnel limits are not exceeded. 
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Within Manpower Plans and Policy Division (MP), additional manning 
controls are implemented to account for the Marine Corps’ patients, prisoners, 
transients, and trainees (P2T2). Accounting for P2T2 against the ASR produces 
the Grade Adjusted Recapitulation (GAR). The manpower metrics specified in 
the GAR serve as the final manpower targets for a specific fiscal year. A 
snapshot of the GARs manpower metrics may be viewed in Appendix B. The 
GAR is the manpower structure document the MSM uses to build its virtual 
architecture from. 

Building off of the absolute structure, personnel movement becomes the 
second critical component to properly forecasting manpower requirements. This 
component is often illustrated in a graphic called the Enlisted Manpower 
Pyramid, shown in Figure 3. It includes external and internal personnel 
movement, which is classified into four categorical areas: accessions/gains, 
retentions/reenlistments, promotions, and losses. 



Figure 3. Enlisted Manpower Pyramid. Adapted from Spafford (2016). 
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These four components yield insight into how Marines are moving through 
the manpower structure. These four components are further broken down to 
capture specific manpower movement. For accessions/gains, sub-categories 
include Marines returning from: 

• Medical Holds 

• End of Active Service (EAS) Holds 

• Deserter Status 

• Fulfilling Recruiter Billets 

• Cross Year Extensions 

• Non-Prior Service Accessions 

For retentions/reenlistments, this category is broken down into two sub¬ 
categories: First Term Alignment Plan (FTAP) and the Subsequent Term 
Alignment Plan (STAP). These plans shape the inventory of the enlisted force by 
controlling the number of enlistments for the respective term force (United States 
Marine Corps, 2010). These plans are critical as they ensure the proper quantity 
of MOSs, with respect to grade distribution, are being retained within the Marine 
Corps. 

The component promotions does not have a specific sub-category, 
however it is proportionally controlled by the Enlisted Career Force Controls 
(ECFC). ECFC impacts two specific areas, first they impact the promotion zones 
control parameters. These controls ensure each Marine receives an opportunity 
to be promoted, as well as ensures a Marine has the proper experience to fulfill 
higher grade responsibilities. Secondly, the controls facilitate the process of only 
retaining the most qualified Marines by forcing Marines who fall below acceptable 
career progression standards out of the service. For example, if Marines are 
continuously being passed over for promotion it is assumed they do not possess 
the high-standard qualifications the Marine Corps is looking for and therefore 
should not continue to be retained within the force (United States Marine Corps, 
2010 ). 
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The component losses has multiple sub-categories, specifically 
categorizing Marines that: 

• Receive commission and become Officers, referred to as an 
enlisted to officer (E20) transition 

• Drop out and separate from Boot Camp 

• Transition into the reserves 

• Reach service limitations 

• Retire 

• Execute general EAS 

• Death cases 

• Non-EAS cases 

Aggregating and associating the four manpower movement categories 
yields detailed rates for Marines moving among the manpower structure. 
Cognizance of these rates allow manpower planners to take today’s force, 
extrapolate the data, and forecast what they believe will be the future’s force. 
Projections are developed at the beginning of each fiscal year to estimate the 
amount of Marines moving within the manpower pyramid in order to anticipate 
the quantity of accessions needed to backfill future vacancies. These forecasts 
allow the Marine Corps to establish milestones that facilitate it to conclude a 
fiscal year exactly at the National Defense Authorization Act’s approved end 
strength. This is the manpower management game Manpower Plans and Policy 
Division is continuously playing, and its success is crucial to ensure the Marine 
Corps’ manpower readiness. 
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II. TECHNICAL REVIEW 


This chapter serves to supply the reader with the technical background 
knowledge that will facilitate the understanding of MSM’s construction. This is not 
an exhaustive background but rather focuses on the key components that give 
the MSM its greatest capabilities. Finally, this chapter will conclude with an 
overview of the scientific methodologies used for the evaluation of the MSM’s 
input and processes sensitivity. 

A. AGENT-BASED MODELING AND SIMULATION 

Prior to agent-based simulation modeling, computer models used 
mathematical formulas to compute statistics of interest (Railsback & Grimm, 
2012). As object oriented computer languages have impacted computer 
programming, its methodology has also impacted modeling and the ability to 
track objects, annotated as agents within simulation, through the system. 

“Agent-based modeling are thus models where individual agents are 
described as unique and autonomous entities that usually interact with each 
other and their environment locally” (Railsback & Grimm, 2012, p. 10). Agents 
are entities that can be constructed to have very specific behaviors. As entities, 
they are able to retain the statistics of their interaction with the system, which 
allows for enhanced data analysis. This includes analyzing how the agent is 
impacted by the rules and processes within the system (Railsback & Grimm, 
2012 ). 

This type of problem framing allows for three separate aspects to be 
examined: the agent as an individual, agents aggregated to form a group, and 
the system as a whole. These aspects facilitate the user’s ability to examine the 
model and the total throughput of an agent within a system, and more importantly 
gain a further understanding of the third, fourth, even tenth-order consequences 
associated with an environment policy decision (Railsback & Grimm, 2012). 
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B. STATES, EVENTS, AND EVENT GRAPH METHODOLOGY 

In modeling and simulation, understanding the throughput of the 
numerous processes and variables that are adjusted within the simulation can be 
a daunting task. To simplify these actions and understand the numerous 
calculations that occur, these processes can be broken down into a few graphic 
illustrations. For a majority of simulations, to include the MSM, the two 
predominant graphic illustrations are state variables and events. 

A state variable is a variable that “has a possibility of changing value at 
least once during any given simulation run” (Buss, 2011, p. 1-1). These items 
provide the user with a snapshot description of the system at any specific point in 
time. Over the course of a simulation, state variables are identified as being 
piecewise constants. This means that a state variable can only change its value 
instantaneously at a specified trigger point. These trigger points can be 
continuous time steps, which is the method of advancement for the MSM, or at 
previously specified non-continuous discrete times. This capability for a state to 
change, to transition to a new state, introduces the constraint called an event 
(Buss, 2011). 

An event is a particular action that occurs within a simulation, and its 
occurrence is the trigger for a state variable to change its value instantaneously. 
Events can be mapped to change a plethora of state variables in a singular 
instance, or as little as none. The key take away is that no state variable is 
allowed to change its value without being initiated to do so by the instance of an 
event (Buss, 2011). 

Various practices have been developed to illustrate these state variables 
and event interactions; one such practice used is the event graph. An event 
graph is an illustration that depicts states, the possible transitions that a state can 
execute, the state variables that are required to be updated with the execution of 
that event, and any constraints that are emplaced for the event to occur. The 
aggregation of these illustrations describes a particular process within the model; 
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and is captured by an event graph (Buss, 2011). Figure 4 is an event graph that 
depicts a specific process; in the case of this example a Multiple Server Queue 
process. The figure is annotated with numerous descriptions to identify the 
various components of an event graph. Dotted red arrows link these annotations 
to their respective event graph illustrations. 


Event Node Edge Transition 



Figure 4. Multiple Server Queue Event Graph with Legend. 

Adapted from Buss (2011). 


Within an event graph, each event is represented by its annotated circle, 
called a node. Directly under the node, the states that are to be updated by the 
execution of that particular event are represented in equation form. Connecting 
nodes to each other is accomplished by a one-way arrow, commonly referred to 
as an edge. Because this edge implicates a transition that can only occur in one 
specific direction, this graph is a directed graph. Resting in the middle of the 
edges are the conditional statements that serve as requirements that constrain 
the next possible event from occurring. These conditions are represented in 
Boolean form; they are either true and allow the transition to occur or they are 
false and disallow it. Located at the beginning of each edge are any parameters 

and arguments that are passed to the next event to specify how it will be 
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executed. These parameters and arguments can serve as metric references that 
affect how certain state functions are executed, or they can serve as dimensional 
parameters that are used to reference specific states in the state transition 
formulas (Buss, 2011). 

Within the context of the MSM, event graphs become extremely useful in 
understanding its numerous processes executed. Specifically, they facilitate the 
ability to easily view the process’s logic, and compare it to actual Marine Corps 
practices. Chapter III documents the structure and main processes within the 
MSM. Specific MSM event graphs can be found there. 

C. RANDOM NUMBER GENERATORS 

The Marine Corps manpower throughput process, the HRDP, is 
fundamentally built off of the individual Marine. As such, while the HRDP is a 
discrete process it naturally must account for the human psychological factors 
that influence every Marine. The Marine Corp’s Doctrinal Publication 1, 
Warfighting, identifies human factors as the intangible dimension of warfare and 
is the root cause of “complexities, inconsistencies, and peculiarities” that shape 
war (MCDP-1, 1997). With respect to manpower, this is most notably their free 
will and the affordability to make decisions that affect their career. These human 
factors can influence a Marine’s decision to reenlist or execute an EAS 
discharge, continue within their MOS or lateral move into a new MOS, or in the 
rare case decide to desert the service. Human factors are also present at other 
key HRDP events. Take for example a promotion board, while the general intent 
is to promote the most qualified Marines, the promotion selection is always 
subject to the human factors of the board and their collaborative decision making 
process. One method to account for the selection and execution of non- 
deterministic events is to incorporate randomness into a simulation model 
(Sanchez, 2007). 

Within the MSM, a Mersenne Twister random number generator is used to 
invoke instances of randomness during the HRDP. Once invoked, the randomly 
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generated number is compared to an event likelihood, which is referenced from a 
database, to determine if the event’s hypothesis is executed (Garrick, 2014). This 
usage of a random number generator gives the MSM its stochastic modeling 
feature. Stochastic, in terms of modeling and simulation, can be defined as when 
“the same set of simulation inputs may produce different outputs” (Sanchez & 
Wan, 2012, p. 2). 

The Mersenne Twister was invented in 1997 by Makoto Matsumoto and 
Takuji Nishimura. It employs iterative possibilities that spread across 623 

dimensions, when using a 32-bit computer operating system (Tia & Benkrid, 
2009). The Mersenne Twister is an incredibly effective random number generator 
and has become the most widely used random number generator in modeling 
and simulation (Tia & Benkrid, 2009). 

Stochastic Modeling provides additional analytic capabilities to the user. 
Unlike deterministic models that yield consistent point output measurements, a 
stochastic model will distribute those measurements over a range. This 
distribution allows for the mean and standard deviation of an output 
measurement to be calculated, which in turn can be derived into likelihood. 
However, in order to produce an effective distribution of output measurements 
the model must be repeated multiple times (Law & Kelton, 2000). Further 
information regarding the implementation and usage of random number 
generators within the MSM can be found in Chapter III. 

D. DESIGN OF EXPERIMENTS 

Design of Experiments (DOE) is a statistical approach in conducting 
experiments to yield information about the processes of a model and how its 
outputs respond to changes in the inputs. The information produced from these 
experiments can identify minimum and maximum thresholds for inputs, capture 
how inputs drive the model to produce its output measurements, as well as 
understand how policies, codified as processes within the model, affect the 
model’s output measurements (Sanchez & Wan, 2012). History has naturally 
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utilized DOE within the agriculture industry; seen countless times in the simple 
scenario of a farmer trying two different fertilizes on his crops to identify which 
produces the best crop yield. With the aid of computer simulation, DOE’s 
capabilities have expanded to examine and evaluate complex system models 
before they are ever implemented into the physical world (Sanchez & Wan, 
2012 ). 

Specific terminology and nomenclature are used to codify critical parts and 
processes of a DOE. The inputs of a system that are to be strategically altered 
per simulation trial are called factors. The number of values a factor can be 
simulated at, also referred as the discrete range of values for a specific factor, is 
annotated as levels. The interaction between the varying input factors and the 
model’s output measures is annotated as the model’s response surface. Lastly, a 
trial instance of the various factors set at specific levels is called a design point 
(Sanchez & Wan, 2012). 


Response surfaces can be expressed mathematically, which in turn can 
be shown as a graphical illustration. These expressions are called metamodels 
and link how the various input factors impact the model’s output measures. How 
input factors effect a model’s response surface can be measured by observing 
two different impact methods: those that show solely main effects and those that 
show main effects with additional quadratic effects (Sanchez & Wan, 2012). Main 
effects are regarded as factors that independently impact the models outputs. 
Specifically, these are the effects captured when a singular factor is altered. 
Quadratic effects reveal how factors impact themselves quadratically, as well as 
how factors interact with other factors in two-way interactions. Equations (1) and 
(2) show the formula representation for main-effects and quadratic-effects, 
respectively (Sanchez & Wan, 2012). 


i=l 


( 1 ) 
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y = A +1AX, +1Aj(X, -X,f + Z t Av(x, -x,)(x, -xj + ^ 

/=1 ;=1 /=1 ;= 1+1 ^ 2 ) 

The final aspect to DOE is the design itself, which defines how the factors 
will be altered per design point for the entirety of the DOE experiment. By varying 
these factors in a strategic way the design can simulate the model over the range 
of possible inputs, called the sample space (Sanchez & Wan, 2012). 

Using DOE’s previous employment in agriculture to illustrate this idea of a 
design, let us create the case study to examine under what conditions will 
produce the greatest crop yield for a farmer. In this experiment, three factors are 
chosen to be varied; seed type, fertilizer type, and pesticide type. Each factor has 
two types of brands that will be used; these are the levels for those factors. 
Listing out every possible configuration yields eight design points for the DOE 
experiment. This type of design is called a full factorial design with three factors 
altered at two levels; it is annotated as 2^ Designs can be illustrated graphically 
or annotated in matrix form to identify the range of possibilities, the sample 
space, that the factors of a DOE can assume (Sanchez & Wan, 2012). Figure 5 
shows a 2^ factorial design in matrix and graphical form. Graphical images are 
great for viewing the sample space and observing how the design points fill it. 
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Figure 5. 2^ Factorial design, graphically and in matrix form, with 
numbered design points. Source: Sanchez & Wan (2012). 
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Filling the sample space with design points introduces the fundamental 
obstacle that must be overcome with DOE designs. On one side of the problem, 
a user wants to obtain detailed information of how a model responds to its inputs. 
However, to obtain a perfect model synopsis a user would need an infinite 
number of design points, which the computational costs to calculate would be 
unfathomable. Conversely, to achieve a low computational cost model a user 
might be sacrificing the level of detail the model’s response surface yields. 

To illustrate this problem better, let’s use the children’s game capture the 
flag. In this game, there are two factors that can be used to evaluate how good a 
child is at playing the game; they are stealth and speed. If the factors are 
restricted to only two levels, the sample space is left fairly open and a child can 
only be categorized into extreme stealth-speed options. However, when 
additionally levels are examined per factor the experimental space is filled with 
more stealth-speed possibilities. This leaves the user with more possibilities to 
classify players, and therefore better identify the specific factor levels where the 
best players resides (Sanchez & Wan, 2012). This effect is illustrated in Figure 6. 



Speed 
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Speed 


Figure 6. 2^ and 11^ Factorial Designs for Capture-the-Flag. Source: 

Sanchez & Wan (2012). 

As seen in Figure 6, being able to better fill the design’s experimental 
space with design points yields substantially more information about a system. 
This technique is called space-filling (Sanchez & Wan, 2012). However, while it 
provides detailed information of a model’s response surface it is extremely 
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computationally expensive to run through all of the specific calculations. 
Comparing Figure 6’s two DOE designs, the 2^ factorial design requires four 
simulation runs where the 11^ factorial design requires 121 simulation runs. One 
solution to lower computational costs while acquiring detailed information of a 
model is through the use of a Nearly Orthogonal Latin Hypercube (NOLH) 
design. A NOLH design accomplishes this by taking a fully filled sample space 
and strategically selecting design points to run computations on. 

Figure 7 depicts this methodical approach pioneered in NOLH designs. 
Specifically, while only a handful of specific design points were selected from the 
original iF factorial design, the output measures provided by the response 
surfaces graph indicate the general areas of peak performance. Additionally, its 
11 computations are a full magnitude lower than the 121 computations required 
in a 11^ factorial design. 
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Figure 7. NOLH Example. Source: Sanchez & Wan (2012). 


Understanding where design points are being placed within a 
experimental space is an important part of any DOE design, especially a NOLH. 
One method to show this layout is through the use of a scatterplot matrix 
(Sanchez & Wan, 2012). A scatterplot matrix is similar to the two-dimensional 
graphs shown in Figure 7, but expands off this to include numerous additional 
dimensions. Figure 8 shows Factorial and NOLH designs and their respective 
abilities to fill the design space. 
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Figure 8. Space Filling Designs. Source: Sanchez & Wan (2012). 

In this thesis, DOE is used to gain insight on the effects of MSM’s inputs 
on its various output measurements, the response surfaces. This experiment 
used 24 factors which were varied at chosen levels to produce a 257 design 
point NOLFI DOE design. A NOLFI design was chosen due to the quantity of 
inputs that drive the MSM, and in real life the HRDP. A more detailed description 
of DOES usage for evaluation and analysis is revealed in Chapter IV. In closing, 
using DOE as an evaluation tool allows a user to identify a model’s inputs cause- 
effect relationship with the model’s outputs, and therefore facilitates the ability to 
derive how the processes in a model are interacting at an abstract level. 
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III. MSM SIMULATION MODEL 


The purpose of this chapter is to annotate the architecture and 
mechanisms that make up the MSM model. We first examine the MSM from a 
high-level approach, starting with the methodology used in constructing the MSM 
and giving an overview of how the program runs. We then move into examining 
the finer mechanisms that drive the MSM. This includes the various modeling 
controls that facilitate the execution of the program, the inputs that set 
parameters and establish data table look-ups, as well as dissecting MSM’s 
simulation processes to show the individual components that drive the model. 
Finally, we examine the outputs the MSM produces. 

A. DEVELOPMENT APPROACH 

The MSM is a computer simulation model, programmed in the Java 
language that uses the capabilities associated with agent-based simulation to 
capture various manpower metrics during its simulation. For the MSM, Marine 
Agents serve as the individual agent instances that navigation through the 
simulation. A bottom up development methodology was used when constructing 
the MSM. With this approach, the developer identified the numerous individual 
manpower processes of the HRDP, built them as Java classes with their own 
state properties and unique behavioral methods, and then aggregated these 
processes to establish the manpower throughput of the FIRDP. 

B. PROGRAM EXECUTION OVERVIEW 

The processes that the MSM uses are aggregated in a specific order to 

accomplish key phases of the MSM program as it runs. At a high-level view, this 

aggregation is organized into five phases. These phases are annotated as the 

user input, database pull, system initialization, simulation, and data output 

processing. The progression of these phases, with the specific processes internal 

to them, is shown in Figure 9 as a flowchart where the rectangles represent 

phases, and the circle nodes inside them represent specific event processes. Out 
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of these phases, the Simulation Phase is the main phase that drives the 
simulation. This phase is responsible for simulating the careers of all the Marine 
Agents, which ultimately gets captured as MSM’s manpower output 
measurements. 



Figure 9. MSM Program Flowchart. Adapted from Garrick (2014). 


The MSM is a time-step simulation where the metric for advancement is a 
one-year increment within the simulation phase. This means that all Marine 
Agents complete a singular process before being shuffled to the next process. As 
such, each Marine Agent is subject to losses, gains, promotions, lateral moves, 
and accessions once per fiscal year before the simulation increments by a fiscal 
year and repeats itself. This process progression can therefore be viewed as a 
waterfall structure; once one process is completed all Marine Agents move 
forward as a group to the next process. The inner components of these 
processes are documented as event graphs in Section E of this chapter. 
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C. MODELING CONTROLS 

Facilitating the execution of the MSM are the simulation controls that 
guide Marine Agents through their simulated careers. These controls include the 
network that Marine Agents use to advance in their careers, the rules and 
conditions that bound the Marine Agent to actual HRDP processes, and the 
usage of randomness to simulate non-deterministic decisions that occur within 
the HRDP. 

1. Structure 

The foundational control mechanism of the MSM is the state graph 
network. This network represents the actual manpower structure of the Marine 
Corps where each state is associated to a direct MOS-Grade pair. This 
architecture was specifically chosen as it mimics the GAR. As previously 
discussed in Chapter I, the GAR is a manpower document that sets target 
quantities for every MOS-Grade pair in order to account for the Marine Corps 
range of manpower specialties and grade-experience while conforming to that 
fiscal year’s end strength ceiling constraint. A snapshot of a GAR can be viewed 
in Appendix B. These MOS-Grade states are connected by a directed graph. 
This directed graph specifies which MOS-Grade state a Marine Agent can 
transition to as it simulates its career. Figure 10 illustrates a snapshot of this 
MOS-Grade state directed graph. Specifically, states are shown as circles and 
the specific MOS-Grade a particular state is representing is annotated in its 
center. Arrows link states to each other, with the arrows direction representing 
the transition direction a Marine Agent can make. When aggregated with all MOS 
and Grade combinations, these states and transitions make up the directed 
graph network for the MSM. With respect to constructing this network, the MSM 
uses a GAR file input to instantiate all the MOS-Grade states for a simulation, 
and a MOS data Excel spreadsheet input to establish the state to state transition 
possibilities. More information regarding these inputs is revealed in Section D of 
this chapter. 
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Figure 10. MOS-Grade State Network Snapshot. 

Adapted from Garrick (2014). 


Figure 10 also shows the progression for MOSs from a recruit to a primary 
MOS. The four categories of MOS progression are: Recruit MOS, Basic MOS, 
Entry Level MOS, and Primary MOS. An important item to note is that the GAR 
input does not specify data for El and E2 grades, which means the MSM does 
not instantiate those Recruit MOS states. To circumvent this data interface the 
MSM translates the distribution of Basic MOSs into probabilities for the E3 grade 
sample space, and then randomly selects a MOS to assign a newly instantiated 
Marine Agent to. This meets the intent of the GAR as greater demand MOSs 
have a higher likelihood of being selected by the MSM’s random number 
generator. Flowever, this initialization for El and E2 grades only occurs once 
when a simulation run is invoked. The remainder of MSM’s program execution 
uses the accessions process to properly simulate for these entry grades. More 
information about the MSM’s processes is discussed in Section E of this chapter. 
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Besides for providing the progression network that Marine Agents 
navigate through, all MOS-Grade states retain specific functionality within the 
MSM. First, they act like a container bin. All MOS-Grade states keep a tally of 
Marine Agents that occupy that state. Conditional rules within the MSM’s source 
code require space availability within a MOS-Grade state for a Marine Agent to 
transition into it. This ceiling is established within the MSM by the GAR input. 
Additionally, MOS-Grade states retain metadata for that state. This data 
eventually becomes the predominance of the output measurements produced by 
the MSM. Metadata that each MOS-Grade state retains includes; 

• Predecessor states that are connected to it. 

• Successive states that a particular state can direct to. 

• Loss and Gain type categories. 

• Ouantity of losses and gains, per HRDP subcategories, captured in 
monthly increments. 

• Retention metrics; to include quantity of Marine Agents eligible for 
retention and quantity of Marine Agents actually retained. 

• Promotion metrics; to include quantity of Marine Agents eligible for 
promotion and quantity of Marine Agents actually promoted. 

2. Deterministic Processing 

Reinforcing the foundational control provided by the MOS-Grade state 
directed graph network are the deterministic processes that drive Marine Agents 
through the simulation. These processes are executed through rule-based 
decisions and are utilized when the outcome of a decision can be simplified into 
a binary option. Boolean conditions are leveraged within if-else statements, 
switch statements, for-loops or while-loops, and many other methods to navigate 
a particular process’s algorithm. This control approach is deterministic because 
an identical Marine Agent, under the same inputs and same parameters, will 
select the exact same edge transitions to navigate down the same career path 
every time. A few examples of binary rules the MSM uses to select edge 
transitions are: 
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• Has the Marine Agent received the maximum allotment of 
promotion Passovers? 

• Can the Marine Agent lateral move to a new MOS? 

• Is there availability within a particular MOS-grade state for a Marine 
Agent to transition into? 

3. Non-deterministic Processing 

Non-deterministic processing can be categorized as the process where 
given the exact same input conditions when repeating a process, a different 
output verdict can be selected (Sanchez, 2007). This is also the process that 
gives the MSM its stochastic modeling functionality. Within the MSM, this is seen 
in situations where identical Marine Agents, under the exact same conditions and 
inputs, will select different edge transitions to navigate down for their simulated 
career. In event graph terminology, this is the process of selecting a transition 
edge that will identify the specific next event that will be executed. 

Non-deterministic processing is used within the MSM to simulate 
decisions a Marine Agent would have to make; an example of such would be 
deciding to reenlist or transition out of active service at the expiration of their 
enlistment contract. Non-deterministic processing is also leveraged to select 
Marines at various boards within the HRDP; an example would be the final 
selection process of selecting Marines to be promoted from a sampling pool filled 
with qualified candidates. Non-deterministic processing is also used to 
incorporate non-idealistic random events that occur within the Marine Corps 
HRDP process; cases of deaths, deserters, or legal holds just to name a few. 

The execution of non-deterministic processing is carried out by employing 
the Mersenne Twister random number generator in two different methods. We 
will classify these two different methods as a percentage line selector and a 
Bernoulli Trial. The percentage line selector incorporates most the characteristics 
of a Markov Chain model, however for only one transition instance per 
invocation. 
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In a percentage line selector, the potential outcomes are categorized and 
bounded on a percentage line that ranges from 0 to 100 percent. The outcomes 
are weighted such that events that are more likely to occur will receive a larger 
range selection on the percentage line. This weighted value is determined by 
historical data that is input into the MSM and translated into a percentage rate 
value. All possible transition events are then positioned on this percentage line 
such that they are all mutually exclusive, they are fully married next to one 
another, and they are normalized to fit the entirety of the percentage line. Finally, 
the random number generator generates a random percentage that will map to a 
singular outcome. This outcome is the decision point for that event and maps to 
the edge transition that particular Marine Agent will navigate down. Figure 11 
illustrates this method. 
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Loss subcategories are not drawn to scale, and their respective values were 
arbitrary created for this illustration. Intent of this illustration is to show how the 
loss subcategories marry up on a number line, that they sum to 100%, and that 
the Mersenne generates a random “pLoss” percentage used to execute one of 
the potential subcategories. 

Figure 11. Loss Subcategories Viewed on Number Line. 

Adapted from Garrick (2014). 


The second non-deterministic processing method used is a Bernoulli trial. 
This method is used specifically within the MSM to simulate the occurrence of 
those non-idealistic events, as well as other unique success and failure cases. In 
this method, a look-up table is referenced to set a threshold that will serve to 
delineate a trail’s success or failure. This success or failure is then mapped to the 
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event decision. The Merseene Twister is again used to generate a random 
number that is translated to a percentage, and then referenced to the Bernoulli 
trial to select a course of action for that singular trail. 

D. MSM’S INPUTS 

The MSM requires numerous inputs in order to run its program. The 
insertion of these inputs occurs at two file/folder locations, where the data 
inserted at these locations are either directly translated to parameters within the 
simulation or are used to reference Marine Corps online manpower archive 
databases. Figure 12 illustrates these inputs in a tree structure. The nodes 
outlined in red illustrate which inputs are directly translated into MSM 
parameters. The nodes outlined in blue illustrate which inputs are used to 
reference online databases. The bullets within the tree show the itemized inputs. 



Figure 12. MSM Inputs Shown in Tree Structure. 

Adapted from Garrick (2014). 
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1. Direct Inputs 

The first location that direct inputs are inserted into is the 
“\\MSM\input\gar_data” folder. Within this folder, the user will manually insert 
GAR files into the folder where each file yields information for a single fiscal year. 
These file names must be changed to follow a standard naming convention of 
“nnenl<FY>.gbr’ and “nnenl<FY>.tre” for the GAR build reports and allocation of 
training reports, respectively, where “<FY>” is replaced with the two-digit fiscal 
year the document represents. There is no limit to the number of GAR files 
inserted into this folder, only that GAR file fiscal years need to match the range of 
years the MSM program will forecast through. Flowever, if the user chooses to 
forecast into the future where no GAR file exists, the MSM will reuse its previous 
FY GAR input for the next simulated year. It is critical that these preparatory 
details are followed as the MSM’s source code is hard-coded to look exactly in 
that folder for exactly those file names. If the file names are in the wrong location 
or labeled incorrectly, the MSM program will not be able to pull the required data 
and construct its MOS-Grade state network, which will therefore crash the 
program. 

The other direct inputs are all inserted into the “siminputs” Excel file. This 
document is broken down into eight sheets, where each sheet is associated with 
a particular HRDP process its inputs will influence. The following sheets require 
values to be input by the user: 

• Main: includes simulation forecast y-years for k-repetitions 

• Career force controls 

• Pay data 

• Monthly boot camp shipment distributions 

• MOS data. This sheet is responsible for providing the series of 
rules that affect Marine Agents and how they transition to the 
various MOS-Grade states 

The Excel file “siminputs” also contains sheets where the user is afforded 

the option to insert non-mandatory inputs. These inputs allow the user to quickly 
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alter some of the simulation parameters to specifically target certain case 
scenarios. Section E of this chapter will go into much further detail, but at an 
overview level, some of the MSM’s Simulation Phase processes use algorithms 
that invoke the functionally to scale the results for a particular process. What this 
means is if the optional user data input is higher or lower than what the MSM is 
using as its baseline value, the MSM will increase or decrease its results, 
respectively. This scaling is also done proportionally to the percentage difference 
between the user input and the MSM’s baseline value. A perfect example of this 
functionality is if a user wanted to re-run a simulation but for a different targeted 
fiscal year end strength. The user could quickly insert that value into the optional 
spreadsheet and receive an ad hoc assessment without having to go through the 
lengthy process of completely rebuilding the GAR files. The sheets these optional 
inputs can be found in are: 

• Fiscal Year Targets 

• MOS Targets 

• MOS and Grade Targets 

Additional details regarding these inputs can be found in Appendix A. 

2. Reference Inputs 

The other type of input that facilitates the MSM’s functions is the reference 
input. These indirect inputs are the historical data the MSM pulls from an online 
Oracle database. Two types of data base pulls occur here: acquiring historical 
data of manpower pyramid transition rates, and inventory. These database pulls 
provide historical data that the MSM uses to compute its various probabilities of a 
Marine Agent executing a specific edge transition. The MSM facilitates the ability 
to pull from two different databases: Total Force Data Warehouse (TFDW) and 
Operational Data Store Enterprise (ODSE). TFDW and ODSE are very similar in 
the data they contain. The TFDW stores historical information at the one-month 
increment. Specific month’s data can be obtained by providing the specific 
sequence number mapped to that month. Conversely, the ODSE maintains real- 
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time manpower metrics vice historical data, therefore providing the user an exact 
inventory of today’s force. The data pulled from these databases includes; 

• Loss data 

• Reenlistment data 

• Promotion data 

• Gains data 

• Recruit loss data 

The other type of database pull is the inventory pull. This pull obtains a 
real sight picture of the manpower layout within the Marine Corps at a specific 
point in time. This means each real historical Marine becomes represented as a 
Marine Agent in order to initiate the MSM. The attributes for each Marine pulled 
are: 

• Primary military occupational specialty (PMOS) 

• Gender 

• Years of service (YOS) 

• Time in grade (TIG) 

• Expiration of current contract (ECC) Year 

• ECC Month 

• Grade Selected For (if pending an approved promotion) 

E. SIMULATION PHASE 

As Figure 9 showed, there are five phases that make up the MSM 
program. The first three phases are responsible for establishing the MSM’s 
environment. This includes reading in user inputs, pulling reference data from 
online data repositories and storing that information within MSM local data tables, 
and then initializing the MSM MOS-Grade state graph network in terms of 
constructing the network and then populating it with the initial distribution of 
Marine Agents. Once these three phases conclude, the program is ready to enter 
the simulation phase. 
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The simulation phase is the main phase of the MSM as it is responsible for 
simulating the specific HRDP processes. This phase simulates for a user 
specified number of years. Specifically, after the last process within the phase 
has concluded, the MSM either will increment to the next fiscal year and repeat 
the simulation phase or, if all forecast years have been simulated, the MSM will 
increment to the data aggregation phase. When simulating a single-year 
iteration, the following manpower processes are executed in the following 
sequence: 

1. Losses/Retentions 

2. Gains 

3. Lateral move with simultaneous promotions 

4. Lateral moves 

5. Promotions 

6. Accessions 

After the sixth and final process, the program stores its calculated metrics 
and determines its next course of action. If there are still additional years left to 
forecast, the program iterates and continues simulating for the following year. If 
the MSM has concluded its final year forecast, the simulation resets all its values 
and will repeat for a user specified number of times. This repetition allows for the 
stochastic properties of the simulation to take effect. 

1. Losses Process 

This process of the simulation model handles the attrition aspect of the 
HRDP. Each Marine Agent within the simulation is pulled and run through a trial 
to determine whether that Marine Agent continues their simulated career or if 
they have been randomly selected to exit the Marine Corps. In the losses 
process event graph, there are two major decisive event-nodes that occur. The 
first examines the Marine Agents years of service (YOS) in a simple if-else 
statement. If the Marine Agent’s YOS exceeds the maximum time in service 

(maxTIS) constraint, as annotated by the user in their ECFC inputs, then the 
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Marine Agent has reached service limitations and is force to exit the Marine 
Corps. If cleared, the Marine Agent continues to the second decision node where 
they undergo a percentage line selector, non-deterministic trial, to determine if 
they will be categorized as one of the possible losses, or if they will continue their 
simulated careers. This trial references MSM’s data tables, which were 
constructed by the reference input-type, to look up historical loss percentages for 
the various loss categories. Data for each loss category is categorized by three 
parameters. The three parameters used are: 

• Grade 

• Inventory type: if the Marine Agent’s ECC is the current simulation 
year, they are annotated as “eas-0.” If the ECC is the following 
simulation year, they are annotated as “eas-1.” And if their ECC is 
beyond that, they are annotated as “neas.” 

• Retirement condition: if the Marine Agent is eligible for retirement, 
they are annotated as “re.” Otherwise, they are annotated as “nre.” 

The percentage is calculated by dividing a loss category by the total 
inventory. Whichever edge condition meets the generated percentage becomes 
the edge transition that is executed. Reference Figure 11 to see an illustration of 
the percentage line selector non-deterministic method. If the Marine Agent 
transitions to a lost event, they are removed from the MOS-Grade state graph 
network, which creates inventory vacancies for that specific MOS-Grade. This 
process is shown in Figure 12 as an event graph. 
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bootP = getBootProb( grade, EAS, ret_EI ) 
retP = getRetjreProb( grade, EAS, ret_EI) 
neasP = getNeasProb( grade, EAS, ret_EI) 
deathP = getDeathProb( grade, EAS, ret_EI) 
e2oP = getE2oProb( grade, EAS, ret_EI) 
otherP = get01herProb( grade, EAS, ret_EI ) 
easP =getEasProb( grade, EAS, ret_EI ) 
reupP = getReupProb( grade, EAS, ret_EI) 
contP = getContProb( grade, EAS, ret_EI) 
rP = getRandomProbO 


Figure 13. Losses Event Graph. Adapted from Garrick (2014). 


Before this process ends, the model checks to see if the optional input 
retention quantity was entered by the user. If so, the program will check to see if 
its original calculation has exceeded this user input threshold. If a zone is under 
its user defined retention zone threshold, no further action will occur. If a zone is 
over its threshold, the program will randomly select a Marine Agent from the 
retention container, re-classify them as EAS, and remove them from the state- 
graph. This itemized removal continues until the specified retention zone is under 
its maximum threshold. 


2. Gains Process 

The Gains Process simulates the real world scenarios when prior service 
Marines return back into the Marine Corps inventory and are counted toward the 
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Marine Corps’ overall end strength. As described in Chapter I, this can occur for 
a variety of reasons; such as returning from recruiting duty or returning from a 
medical hold. This process is executed by a Bernoulli, non-deterministic, trial. 
Specifically, every time a Marine Agent is tried in the losses process, the gains 
process is invoked and ran in parallel. One Bernoulli trial is run for each possible 
gain category (gainCat) per each process invoked. Look up tables, which were 
constructed by the MSM’s reference inputs, yield the trial success probabilities 
for each gainCat. Three parameters are used to determine the specific 
percentage to use for each trial: 

• gainCat 

• Grade 

• Base inventory for that grade 

The percentage is calculated by dividing the quantity of a particular 
gainCat for the identified grade by the base inventory for that grade. Figure 14 
displays the event graph for the Gains Process. 
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Figure 14. Gains Event Graph. Adapted from Garrick (2014). 

If the Bernoulli trial was successful, the MSM will instantiate a new Marine 
Agent and insert them into the MOS-grade state network. To accomplish this, the 
MSM randomly generates all required Marine Agent attributes which include: 

• YOS 

• Time in grade (TIG) 

• ECC 

• PMOS 

• Grade 

• Month gained 

3. Lateral Move Promotions Process 

The Lateral Move Promotions Process is the third process invoked during 
the Simulation Phase. It combines the lateral move process, where a Marine 
Agent obtains a new MOS within the same grade, with the promotions process. 
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where a Marine Agent obtains a higher grade but within the same MOS. This can 
be summarized as a simultaneous horizontal and vertical move within the MOS- 
Grade state graph network. This process is unique in that it only targets the small 
population of Marine Agents with MOSs that allow for career progression to new 
MOSs, lateral moves. This rule constraint is constructed from the direct input 
“FY<FY> MOS data,” which is an Excel sheet that resides within the “siminputs” 
Excel document. 

The lateral move promotions process was strategically placed to precede 
the regular lateral move and regular promotion processes due to the inventory 
quantity of Marine Agents they each contain. This means Marine Agents that 
have the potential for a new MOS are given the opportunity to fill vacant billet 
spots in other MOS-Grade states before Marine Agents with standard career 
progressions. If these processes were reversed there would be a much greater 
probability that all potential MOS-Grade states would be filled by standard career 
progression Marine Agents, leaving lateral move promotable Marine Agent 
candidates stagnant. 

The Lateral Move Promotions Process is also the first process where the 
procedural priority for processing Marine Agents is critical. This stems from the 
constraint that a state must possess billet vacancies in it to allow a Marine Agent 
to transition to it. To accommodate this, the MSM examines Marine Agents in 
reverse grade order; starting with the grade E9 and then working downwards. 
This allows Marine Agents to move into higher billet vacancies immediately, 
which therefore creates a billet vacancy in the MOS-Grade state they just left. 
This effect trickles downward to the lowest grade of Marine Agents. 

The algorithm for the lateral move promotions process contains five steps. 
Figure 15 shows this process in event graph form. In procedural order, the 
algorithm’s steps are annotated as: 

1. Select a MOS-Grade state for Marine Agents to lateral move and 
promote into. 
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2. Build a list of Marine Agents that are allowed to be laterally moved 
and promoted into the identified MOS-Grade state. 

3. Count the number billet vacancies within the targeted MOS-Grade 
state. This is accomplished by obtaining the delta between current 
occupancy and maximum allowed occupancy as dictated by the 
GAR input. 

4. Check if the optional input “End Strength” was entered by the user. 
If it was, the MSM will take the percentage difference between the 
GAR’S total strength and the user’s optional input End Strength, 
and apply that percentage to the MOS-Grade states to increase or 
decrease the total amount of billet vacancies respectively. If the 
user did not enact this option, the program will use the number of 
billet vacancies found in step 3. 

5. While billet vacancies and Marine Agents exist, randomly select a 
Marine Agent to execute the lateral move and promotion transition. 


(next toGrade exists) 


(next toMos exists) 


(next toMos exists) 



eligibleList = buikJLisU toMos, toGrade) marine = eligibleList.random() nodeVac = nodeVac -1 

nodeVac = garVacancies (toMos, toGrade) 


Figure 15. Lateral Move Promotion Event Graph. 

Adapted from Garrick (2014). 


4. Lateral Move Process 

The lateral move process follows the exact same methodology as the 
lateral move promotions process except for two critical areas. First, the process 
will only move Marine Agents horizontally to a new MOS; so the Marine Agent 
will maintain their same grade. Secondly, this process incorporates P2T2 
possibilities into its algorithm. Specifically, being selected by the board does not 
guarantee the Marine Agent will transition to their updated MOS-Grade state. 
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Instead, the algorithm simulates whether or not the Marine Agent will be placed 
back in the state-graph or if they will be placed into the P2T2 container for that 
fiscal year. This decision is made by referencing the length of the school, time to 
train MOS metric, the Marine Agent would have to attend to learn their new trade. 
If the school is longer than 365 days, one-year, the algorithm directly places the 
Marine Agent into the P2T2 container since they will be without a doubt still in 
school when that fiscal year ends. If the school length is shorter, the algorithm 
calculates the probability of being at that school at any particular point in the 
fiscal year where the probably is the duration of the school divided by 365 days. 
Once calculated, this percentage is passed to a Bernoulli, non-deterministic, trial 
where one trial is run. If the Bernoulli trial was successful, then the model will 
place the Marine Agent into the P2T2 container. Otherwise, they will return to the 
state-graph at their updated MOS-Grade state. Figure 16 shows this process as 
an event graph. 



Figure 16. LatMove Event Graph. Adapted from Garrick (2014). 

5. Promotions Process 

This process compliments the previously stated lateral move promotions 

process in that it is the mutually exclusive counterpart to the lateral mover 

indicator. Specifically, this process focuses promoting Marine Agents who do not 
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possess MOSs that allow for lateral moves. Non-lateral move MOSs comprise of 
approximately 95% of the total MOS inventory (Spafford, 2016). Therefore, it is to 
no surprise that the algorithm steps and event graph, shown in Figure 17, are 
identical. 


(next toGrade exists) 


(next toMos exists) 


(next toMos exists) 



Figure 17. Promotions Event Graph. Adapted from Garrick (2014). 

6. Accessions Process 

The accessions process is the final process of the simulation phase and is 
predominately summarized as the overall backfill of all MOS-Grade state 
vacancies that have trickled their way down to the junior grade levels. The 
algorithm is conducted by executing the following steps. 

1. Set baseline accession targets per MOS: This is accomplished by 
aggregating the current inventory in the MOS-Grade state graph 
network for grades El through E3, categorizing by MOS. It then 
subtracts that MOS’s entry level aggregate from the GAR’s allowed 
quantity, which is the GAR column labeled “E3.” The initial 
aggregation of the MSM’s current entry level inventory is required 
due to the data interface constraint with the GAR; that is the GAR 
does not identify MOS quantities for grades El and E2. 

2. Distribute MOS targets to entry level grades: By default, the MSM 
takes step Ts deliverable and distributes them to the paygrades 
El, E2, E3, within that MOS, by the constant ratio 75.84%, 24.15%, 
and 0.01%, respectively. Flowever, if the user set the optional MOS 
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accession targets in the siminputs Excel document, the paygrade 
distribution will be applied to this quantity vice step 1 ’s deliverable. 

3. Set MOS-Grade male and female targets: The MSM takes step 2’s 
deliverable and then divides that quantity into male targets and 
female targets. By default, this distribution is calculated by the 
MSM’s reference input from the historical Marine Corps Oracle 
online databases. However, if the user sets the optional female 
accession targets in the siminputs Excel document, the 
male/female distribution will then be proportionately adjusted to 
match the user’s target. 

4. Instantiate a Marine Agent; For every male/female MOS-Grade 
target, the MSM will instantiate a new Marine Agent. While the sex, 
MOS, and grade are already determined, the MSM will use its 
percentage line non-deterministic selector to randomly determine 
the contract length of the Marine Agent and the Month in which that 
Marine Agent entered boot camp. Contract length probabilities are 
set by the required user input ‘FY<FY> MOS Data’ sheet in the 
Excel file “siminputs.” 

5. Test for boot camp loss, and insert into MOS-Grade state graph 
network: Each instantiated Marine Agent will be passed through a 
boot camp losses trail. This trial is very similar to the Losses 
process event graph. If a loss event is not selected, then Marine 
Agent is then inserted into their respective MOS-Grade state. 

F. MODEL OUTPUTS 

At the completion of the Simulation Phase, the program records the data 
associated with that particular repetition run. The program will then iterate and 
run the entire simulation over again, using the same inputs, to obtain new output 
metrics. This repetition allows for stochastic properties to take effect. Once all 
repetitions have run, the program computes a statistical analysis of the different 
runs. This is called the data aggregation process. Each metric within the process 
is analyzed by its: 

• Means 

• Standard Deviation 
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Maximum 


• 

• Minimum 

These output measures are aggregated at three different tiers, where 
each tier yields the same data but at different levels of fidelity. Tier I data is the 
most abstract data output, where the data obtained from all the MOS-Grades are 
aggregated into a fiscal year sum. Tier II still aggregates its data into fiscal year 
sums, however it incorporates three possible classifiers the data can be sorted 
into: MOS, Grade, or MOS and Grade. Tier III expands off of Tier II in that it 
keeps the same classifiers, but the data drills into the monthly level. Figure 18 
depicts this layered metric tier, and its respective scope of detailed information 
the Model’s user is wishing to obtain. For Chapter Ill’s Design of 
Experimentation, Tier I output data was utilized. For Chapter IV’s Validation, a 
combination of Tier I and Tier II, with categorization by grade, output data was 
utilized. More information regarding the output measures, and their respective 
definitions and metric scope, is listed in Appendix D. 


USMC Whole 
Aggregated by Year 

Tierl — 


Aggregated by Year Aggregated by Year Aggregated by Year 
Tier 2 Categorized by MOS Categorized by Grade Categorized by MOS & Grade 


Aggregated by Month Aggregated by Month Aggregated by Month 

Categorized by MOS Categorized by Grade Categorized by MOS & Grade 


Figure 18. Depiction of MSM’s Output Metrics, Aggregated at Different 

Levels. Adapted from Garrick (2014). 
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IV. DESIGN OF EXPERIMENTS EVALUATION 


This chapter describes the design, procedure, and experimental results 
used for conducting the design of experiments (DOE) evaluation. This evaluation 
provided two key metrics; it verified the input-to-output response correlation, and 
it identified the input significance priority. Correlation is the process of looking at 
two variables and determining if one influences the other. This influence can be 
either strong or weak and incorporates the direction, positive or negative, toward 
which the variables will react (Law & Kelton, 2000). This strength and direction 
relationship is often referred to as the correlation gradient (Law & Kelton, 2000). 
Within this DOE evaluation, gradient was the primary metric for measurement 
used as it allowed for cause-effect HRDP relationships to be easily identified and 
analyzed. 

The second evaluation metric was the input significance priority. Input 
significance priority is the nomenclature used to identify which terms were 
responsible for having the most impact on the produced metamodel. This is 
determined by computing the ratio between the parameter coefficient within a 
metamodel’s response surface equation and its standard error (“Estimates,” n.d.. 
Sorted Estimates section). Terms with a higher level of significance identify which 
inputs are more sensitive in forecasting future data predictions. 

To compute statistics from the DOE experiment, we used the statistical 
software suite JMP Pro 12. An overview of the functionality utilized is described 
in Section B of this Chapter. 

A. CONSTRAINTS AND DESIGN 

This section describes the constraints that were presented in subjecting 
the MSM to a DOE. It highlights the decisions made to comply with these 
constraints, to include what the final DOE design entailed. 
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1 . 


Factor Selection 


DOE is naturally limited by the number of factors it can take in and create 
a design from. This limitation stems from the computational costs required in 
conducting an experiment with a large set of factors while simultaneously 
maintaining the experiment’s orthogonality. The largest NOLH that exists today 
allows for up to 29 factors, which produces a 257-design point design (Cioppa & 
Lucas, 2007). A Nearly Orthogonal Balanced design is a different type of DOE 
design that does allow for more factors to be used (Vieira Jr. et al, 2012). 
However, this design incorporates a mix of discrete and continuous factors that 
did not interface well with the MSM and therefore was discarded. 

Condensing MSM’s numerous inputs to map to 29 DOE factors proved 
challenging. This problem stemmed from MSM’s multi-dimensional look up tables 
that are constructed from its reference input Oracle database pulls. Take for 
example the loss rate inputs for just the loss subcomponent EAS. This 
subcomponent takes the parameters of MOS, Grade, and Retirement Eligibility to 
look up the specific EAS rate within those dimensions. Just those EAS loss rates 
yield 4320 inputs; 240 MOS x 9 Grades x 2 Retirement Options = 4320 inputs. 
Extrapolating this multi-dimensionality across the other HRDP process categories 
unquestionably exceeds a NOLH factor threshold. To circumvent this constraint, 
we implemented the use of abstraction within our DOE. 

With the abstraction methodology, we targeted the specific subcomponent 
of a HRDP process as a single DOE factor vice assigning a factor to every input 
in its multi-dimensional table. With this approach, the DOE factor acted as a 
coefficient that adjusted the subcomponent it was representing. Because this 
subcomponent was uniformly adjusted every time a value was returned from the 
look up tables, the orthogonality of the experiment was maintained. 

Table 1 illustrates the application of this abstraction methodology. 
Specifically, the table provides a snapshot of the “Continue Rate” category, which 
is a subcomponent of the losses HRDP process. As the table shows, its multi- 
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dimensionality yields six inputs for just a single grade return. The original inputs 
are annotated in the column “Continue Rate” and serve as the baseline. The 
column “NOLH Factor” reflects what that DOE factor is set at for that particular 
design point. Acting as a coefficient, the “NOLH Factor” is multiplied with the 
baseline to produce the adjusted input; labeled “Function Rate.” This adjusted 
rate is what is returned to the function, allowing that HRDP process’s algorithm to 
continue its execution in alignment with the DOE design point. 


Table 1. Snapshot of Factor Adjust Rates. 

Adapted from Garrick (2014). 


Grade 

Retirement 

Condition 

Inventory 

Condition 

Continue Rate 

NOLH 

Factor 

Function Rate 

E5 

nre 

neas 

96% 

96% 

92% 

nre 

eas-0 

21% 

96% 

20% 

nre 

eas-1 

87% 

96% 

84% 

re 

neas 

100% 

96% 

96% 

re 

eas-0 

67% 

96% 

64% 

re 

eas-1 

100% 

96% 

96% 


Loss rates are pulled from the Marine Corps online Oracle database and stored 
locally within the MSM program for the simulation. Local data tables contain three 
dimensions: grade, retirement condition, and Inventory condition. Retirement 
condition uses the codes “re” and “nre” to Identify the parameters of being 
retirement eligible and not retirement eligible, respectively. Inventory conditions 
use the codes “eas-0,” “eas-1,” and “neas” to Identify a Marine EASIng within the 
current fiscal year, EASIng during the next fiscal year, and EASIng beyond two 
years out, respectively. 

After condensing MSM’s inputs, a total of 24 factors were chosen to adjust 
the various inputs of the MSM. These factors are annotated in Table 2. The 
factors labeled “Loss_” or “Gain_” were factors that received the abstraction 
method. All other factors were direct replacements of MSM inputs. The columns 
“Low Value” and “High Value” indicate the lower and upper bounds design points 
could range within. The column “Decimals” indicates the number of decimal 
placeholders a design point’s precision was set to. 
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Table 2. DOE Factors and Their Ranges. Adapted from Garrick (2014). 


Factor Name 

Factor Description 

Low 

Value 

High 

Value 

Decimals 

TISmin 

Minimum Time-In-Service (TISmin) required for a 
Marine Agent to be promoted to the superior 
targeted grade. 

-2 

3 

0 

TISmax 

Maximum Time-In-Service (TISmax) allowed for a 
Marine Agent to remain within its current grade. 

-4 

4 

0 

TIGMin 

Minimum Time-In-Grade (TIGmin) required for a 
Marine Agent to be promoted to the superior 
targeted grade. 

-1 

3 

0 

PromTgt 

Promotion target (PromTgt) is the desired 
cumulative years of service to achieve a targeted 
grade. 

-2 

4 

1 

VIZmin 

Variable In-Zone (VIZ) minimum is when mean 
promotion time for targeted grade is within limits, 
this standard selection opportunity is used. 

-0.1 

0.05 

3 

VizStd 

Variable In-Zone (VIZ) standard is when mean 
promotion time for targeted grade is too slow, 
using VIZ minimum opportunity allows Marine 
Agents to compete earlier in their career, speeding 
up average promotion timing. 

-0.05 

0.05 

3 

VIZmax 

Variable In-Zone (VIZ) maximum is when mean 
promotion time for targeted grade is too fast, using 
VIZ maximum opportunity reduces the number in 
zone, delaying opportunity to compete for junior 
Marines thereby slowing the average time in 
service for promoted Marines. 

-0.05 

0.1 

3 

PassO 

Passover (PassO) specifies the maximum number 
of passovers a Marine Agent may have to remain 
within its current grade. 

-2 

2 

0 

ReThsh 

Reenlistment threshold (ReThsh), if passover limit 
is reached. Marine agents may not reenlist beyond 
this threshold. 

-2 

2 

0 

Loss_Boot 

The loss rate at which Marine Agents exit the 

Marine Corps during bootcamp. 

0.6 

1.1 

3 

Loss_Retired 

The loss rate at which Marine Agents exit the 

Marine Corps due to retiring. 

0.6 

1.1 

3 

Loss_NEAS 

The loss rate at which Marine Agents exit the 

Marine Corps with a Non-EAS separation code. 

0.6 

1.1 

3 

Loss_Death 

The loss rate at which Marine Agents exit the 

Marine Corps due to death. 

0.6 

1.1 

3 

Loss_e2o 

The rate at which Enlisted Marine Agents receive 
an enlisted to officer (e2o) commission, and 
therefore leave the enlisted manpower structure. 

0.6 

1.1 

3 

Loss_Other 

The loss rate at which Marine Agents exit the 

Marine Corps due to other means not documented 
with a specific separation code. 

0.6 

1.1 

3 

Loss_EAS 

The loss rate at which Marine Agents exit the 

Marine Corps due to the end of their enlisted 
contract called End of Active Service (EAS). 

0.6 

1.1 

3 

Loss_Reup 

The rate at which Marine Agents reenlist. 

0.6 

1.1 

3 
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Factor Name 

Factor Description 

Low 

Value 

High 

Value 

Decimals 

Gain_EAD 

The rate at which Marine Agents return from their 
extended active duty (EAD) recruiting tour. 

0.6 

1.1 

3 

Gain_Desert 

Marine Agents that return after having become a 
deserter. 

0.6 

1.1 

3 

Gain_Other 

Marine Agents that return to service for other 
means not captured. 

0.6 

1.1 

3 

Gain_OCC 

The rate at which Marine Agents return to service 
with a OCC gain code. 

0.6 

1.1 

3 

Gain_Cont 

The rate at which Marine Agents return to service 
with a Continuous gain code. 

0.6 

1.1 

3 

Gain_Rever 

The rate at which Marine Agents return to service 
with a Reversion gain code. 

0.6 

1.1 

3 

Gain_Broken 

The rate at which Marine Agents return from 
medical holds. 

0.6 

1.1 

3 


After solidifying the DOE factors, the actual DOE design was constructed. 
We used the Naval Postgraduate School SEED center’s open source document, 
titled “NOLHdesigns_v6.xls, to construct our NOLH design of 24 factors and 
257 design points (Sanchez, 2011). This open source document uses MACROS 
and hidden formulas within its template to automatically compute design point 
values for the design. The full design can be found in Appendix E. 

To examine the design’s orthogonality, we constructed a correlations plot 
by using JMP’s multivariate methods functionality. The correlations plot illustrates 
how each NOLH factor relates to other factors. If the design was completely 
orthogonal, each comparison would have a value of zero. However, because the 
NOLH is a nearly orthogonal design, some relationships do exist. From the 24- 
factors selected as inputs, the top three correlations were: 

• 0.0876 (TISmin with TIGmin) 

• 0.0451 (TIGmin with GainOther) 

• 0.0369 (Passovers with LossOther) 

The full correlations plot of the NOLH design can be found in Appendix D. 
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2. Response Surface Selection 

The MSM produces 71 different output metric categories. In theory, a 
response surface could be calculated for each MSM output metric, but that level 
of fidelity exceeded the scope of this thesis. Therefore, we used an aggregation 
methodology to analyze a HRDP process as a whole vice drilling into the 
individual subcomponents’ response surfaces. This aggregated approach was 
applied to three HRDP process: 

• Losses 

• Gains 

• Retention Zones A-SR 

Other output metrics yielded data that was already holistic to a HRDP 
process and did not require aggregation. The four response surfaces that were 
measured with this approach were; 

• Promotions 

• LatMoves 

• End Strength 

• Patients, Prisoners, Transients, and Trainees (P2T2) 

After selecting which output metrics we would compute response surfaces 
for, we then had to select what dimensionality of the output data sets we would 
analyze. As described in Chapter III, MSM’s outputs allow the user to view 
forecasted manpower data either at an aggregated, holistic Marine Corps, level 
or it can drill down into the MOS-Grade level. Additionally, these data values can 
be viewed at monthly intervals or at an aggregated fiscal year level. To meet the 
intent of this thesis, we choose the data set that was holistic to the Marine Corps 
and at the fiscal year interval. 

Additionally, because the MSM allows a user to specify how many 
forecast years it will simulate to and for how many repetitions, we had to isolate 
these parameters too. Ultimately, we decided the fourth forecast year’s data 
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output would be analyzed because that is generally the first opportunity when 
enlistment contracts end. As for repetitions, to yield an appropriate distribution of 
MSM’s stochastic capability, it was decided that each NOLH design point would 
be replicated 30 times. The average value from the replicated data sets was then 
used for evaluation. 

3. Modeling Loss Constraints 

As described in Chapter three, the losses process uses the percentage 
line non-deterministic method to select which loss event a Marine Agent will 
execute. Figure 19 illustrates this type of method. The method is predicated on 
the fact that the rates of all loss events are normalized onto the percentage line 
such that they always aggregate to a full 100% value. As a reminder, eight loss 
events and one continue simulation event make up the list of possible transition 
events a Marine Agent can execute: 

• Bootcamp Loss Event 

• Retirement Loss Event 

• NEAS Loss Event 

• Death Loss Event 

• Enlisted to Officer Commission Loss Event 

• Other Separations Code Loss Event 

• EAS Loss Event 

• Reenlistment Loss Event 

• Continue Simulated Career Event 

Normalization is ensured due to the structure of the supporting data tables 
and how the computations source code was written in the algorithm. Specially, 
the data look-up tables organize its data such that the losses table contains only 
the data used for computing the loss events. This ensures all loss events are 
mutually exclusive, and the entire loss-data sampling space will be used for 
processing one Marine Agent. This inherent setup becomes problematic for the 
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DOE experiment because the DOE Factors will adjust the rates of the loss 
events independently, and therefore, may yield DOE design points where the 
aggregation of loss events exceeds, or falls short of, the intended 100% value. 



Random Number Generator 



o 

o 

o 



“pLoss" 


V 


/ 


0% 3% 10% 18% 19% 30% 32% 64% 80% 100% 


Boot 

Retire* 

NEAS 

Death 

Other 

Continue 

EAS 

Reenlist 

Loss 

ment 

Loss 

l.oss 

Loss 

Simulated Career 

Loss 


3% 

7% 

8% 

1% 

2% 

32% 

16% 

20% 


Loss subcategories are not drawn to scale, and their respective values were 
arbitrary created for this illustration. Intent of this illustration is to show how the 
loss subcategories marry up on a number line, that they sum to 100%, and that 
the Mersenne Twister generates a random “pLoss” percentage used to execute 
one of the potential subcategories. 

Figure 19. Percentage Line Non-deterministic Method. 

Adapted from Garrick (2014). 


After analyzing the constraint, three courses of actions presented 
themselves for how to proceed. The first option was to exclude the Losses 
Process from the NOLFI design, and therefore not evaluate the process. This 
option was discarded because the loss function is one of the most important 
functions of the model. This is because the model tries to maximize its force 
structure by adding Marine Agents into the MOS-Grade state graph network but 
is limited by the constraint that there must be available vacancies in a MOS- 
Grade state for a Marine Agent to move into it. Therefore, if Marine Agents do not 
vacate a MOS-grade state the model becomes stagnant. 

Option two was to include all possible transition events, allowing the 
NOLH design to adjust them independently, and then normalize them prior to 
running the execution statement block so they always summed to the full 100% 
value. This option was discarded as it greatly decreased the orthogonality 
between the factors. 
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The third option was to focus on the eight subcomponents that comprise 
the loss category, adjust each of them independently as the NOLH design 
dictates, and then use the ninth subcomponent “Continue” to absorb any 
remaining difference in the number line so that the statement block still sums to 
100%. While the “Continue” subcomponent would no longer remain independent, 
the objective of the experiment would still be achieved because all the 
subcomponents that capture loss would still be varied independently, and 
therefore their inputs would be independently captured in loss’s response 
surface. To ensure a “Continue” absorbing event had available space to exist on 
the percentage line, the DOE loss factors were bounded from 60% to 110% of 
their original baseline values. While this factor range focused on lower rate 
values, orthogonality of the experiment was retained with this option, and 
therefore still yielded accurate metamodels for this experiment. This was the 
option chosen to comply with the loss process’s constraints. 

4. Scaling Constraints 

As discussed in Chapter III, the MSM possesses the capability for a user 
to insert optional inputs. These optional inputs allow a user to build a specific 
case study to be simulated, or simply allows for ad hoc simulation adjustments. 
However, these optional inputs invoke scaling algorithms that sit on top of the 
Simulation Phase’s processes. When these scaling algorithms are invoked, 
orthogonality of the DOE is not maintained since the values are being adjusted 
independently of the DOE’s design points (Sanchez & Wan, 2012). Therefore, 
this DOE refrained from entering any values into the optional inputs. 

B. PROCEDURES 

1. Experiment 

This experiment required two efforts to be executed. The first effort was to 

establish the baseline MSM inputs. This would be the baseline that the DOE 

factor’s design points would adjust from. We set up the MSM to simulate four 

years, for 30 repetitions. For the reference inputs, we used the end of fiscal year 
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2013 data to initialize the MSM. The initial forecast year was therefore 2014. As 
previously annotated in Section A, no option inputs were inserted in order to 
refrain from invoking the scaling algorithms of the MSM. Table three illustrates 
the inputs inserted into the “Distributions” sheet of the input file “siminputs.” 
Figure 20 illustrates the baseline Enlisted Career Force Controls (ECFC) used. 
Lastly, the “siminputs” Excel sheets “Base Pay” and “MOS Data” used the 
configuration data approved for fiscal year 2014, respectively. 


Table 3. New Recruit Shipping Distribution DOE Inputs 


Oct 

Nov 

Dec 

Jan 

Feb 

Mar 

Apr 

May 

Jun 

Jul 

Aug 

Sep 

6.71% 

7.93% 
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Figure 20. Enlisted Career Force Controls. Adapted from Garrick (2014). 


The second experimental effort was the DOE experiment itself. To 
execute the DOE, we created an additional Java main class. The purpose of this 
test-main class was to control the DOE experiment while ingesting the next 
design point’s inputs, running the MSM program for that design point, and then 
extracting the associated forecasted data. The general algorithm used by the 
test-main’s source code was as follows: 

1. Extract NOLFI design point inputs. 

2. Update MSM’s input document “siminputs” with the respective 
design point values. 
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3 . 


Invoke the main MSM program to run the simulation with the 
updated inputs. 

4. Extract and record the output results. 

5. Repeat with the next design point’s inputs. 

2. Analysis 

To compute statistics from the DOE experiment, we used the statistical 
software suite JMP Pro 12. Specifically, the software provides a function called 
Fit-Model. With this function, the user selects the dependent and independent 
variables to run the JMP function. The dependent variable was the response 
surface of the particular metamodel being evaluated. The independent variables 
were the DOE input factors used in the experiment. JMP software uses the term 
“role variable” to identify the response surface and “model effects” to specify the 
relationship of how the independent input factors will be analyzed. Recalling from 
Chapter II, a NOLH design allows for main effects, two-way interactions, and 
quadratic effects to be examined. Within the JMP software, the DOE factors were 
used to construct the “model effects” by importing them as two groups, which 
produced a total of 324 DOE terms to be analyzed: 

• Polynomial to 2nd Degree construction was used to view the main 
effects and quadratic effects terms for all the DOE design factors. 

• Factorial to 2nd Degree construction was used to view all the two- 
way interaction terms of the model’s effects. 

Initially, JMP’s stepwise regression was run to determine which input 
factors were selected as having a significant impact on the response surface of 
the metamodel. To accomplish this, JMP uses a Bayesian information criterion 
(BIG) for identifying significant terms. The mathematical formula used to compute 
BIG is shown in Equation (3), where k is the number of estimated parameters in 
the model and n is the number of observations in the data set (“Likelihood, AIGc, 
and BIG,” n.d.). This initial computation of significant terms reduced the number 
of terms to be analyzed by approximately 90%. 

BIC = -2 log Likelihood + k ln(n) (3) 
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With the updated significant terms selected, JMP’s Fit-Model was run a 
second time using a Standard Least Squares regression. This final regression 
produced seven statistical reports that enabled the analysis of each metamodel: 

• Effect Summary: This report yields the “LogWorth values for the 
effects in the model” (“Effect Summary Report,” n.d.). The 
mathematical formula to compute LogWorth is shown in Equation 

(4) , where p is the term’s p-value. 

LogWorth =-log^^ip) ^ 4 ^ 

• Actual by Predicted Plot: This plot visually shows the reader how 
well the produced metamodel fits the input data. In the case of our 
DOE experiment, the input data are design points of the NOLH. 
The actual versus predicted values are a comparison of what the 
model actual yielded during its stochastic computations compared 
to what the metamodel’s response surface equation calculated. 

• Summary of Fit. This statistical report provides numerous values 
which yield a statistical overview of the model. For this evaluation 
the adjusted R-Square value was used to normalize and compare 
the metamodels against one another because each model could 
have different numbers of identified significant terms. This number 
ranges from 0 to 1, where the closer the value is to 1 the better the 
model’s fit is to the data. Its specific formula is shown in Equation 

(5) ; where MS is the means square from the model’s source Error, 
the source of the sum of squares is the corrected total (C. Total), 
and DF is the degrees of freedom from the model’s corrected total 
(“Regression Reports,” n.d.. Summary of Fit section). 

Rsquare _Adj = l -- (5) 

SumOfSquares / DF 

• Analysis of Variance. This report yields model calculated values, 
such as the mean square for the metamodels error used in 
Equation (5). 

• Parameter Estimates. This report shows the estimates for all terms 
post BIC selection. These estimates represent the coefficients used 
in the response surface equation. This report also includes the 
Intercept estimate, which is referring to the intercept coefficient for 
the response surface equation. 

• Sorted Parameter Estimates. This report shows the parameter 
estimates for each term, where each term is sorted in a decreasing 
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order of significance. Significance is determined by the absolute 
value of the calculated “t Ratio,” which is the ratio of the estimate 
when compared to its standard error (“Estimates,” n.d.. Sorted 
Estimates section). 

• Prediction Profiler. This report yields a visual that allows the user to 
gauge the sensitivity of the metamodel’s terms, that is the response 
surface’s gradient. As a quick reference, profiles that have a more 
horizontal orientation have little impact on the metamodel’s output 
measurements as the term changes. Conversely, profiles that have 
a more vertical orientation are considered to be sensitive because a 
change in its value can cause a significant change in the 
metamodels output measurements. 

Lastly, all the significant terms from all the metamodels were aggregated 
and analyzed to determine which terms had an overall significant impact on the 
MSM. 

C. ANALYSIS 

This experiment was computationally intensive. Each design point took an 
average of 16 minutes to run, utilizing an Intel Core i7 processor with a 16 GB 
RAM. Extrapolating that timeline over 257 design points, this experiment took 
approximately 2.85 days to complete. 

1. Losses Metamodel 

The first metamodel evaluated was Losses. The response surface metric 
was constructed by aggregating all the loss metrics output by the MSM. 
Specifically, this included the loss categories with Non-EAS and EAS separation 
codes. Each separation code has the following sub-categories: NEAS, deaths, 
E20, retired, EAS, bootcamp, and other. Reference Appendix C for more 
information regarding MSM’s output metrics. 

Figure 21 shows the seven JMP statistical reports for this metamodel. This 
model had an excellent fit, indicated by its 0.98778 adjusted R-Square value. 
This is visually reinforced with the “Actual by Predicted Plot,” where the model’s 
response surface equation predicted a forecast that was within 1.3% of the actual 
simulated output measurements. 
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For our first metric, correlation, the metamodel yielded very promising 
information. A visual of this correlation is shown in the Prediction Profiler report; 
where it plots the profile for the six factors that yielded the greatest effects. From 
initial inspection, the six factors all had gradients that coincide with the Marine 
Corps’ FIRDP process. The two greatest factors were loss due to end of active 
service (EAS) and reenlisting (Reup). For EAS, the profile shows that as the rate 
of EAS increases, so does the quantity of Marine Corps total losses for a fiscal 
year, which is as expected. The profile also indicates an exponential relationship. 
Specifically, lower EAS loss rates have a greater sensitivity in predicting overall 
loss quantities. 

Conversely, the Reup’s loss rate had a linear relationship in the negative 
gradient direction. This directly aligns with the HRDP because if more Marines 
are staying in the Corps for a subsequent term enlistment then there are less 
Marine Agents that could exit the Corps. The other rates of non-EAS and 
bootcamp losses yielded gradients that had the correct slope and magnitude, 
both of which had less of an impact on the over loss quantity. 

A factor that had an unanticipated significance in the Losses metamodel 
was the minimum time in grade (TIGmin) factor. As Figure 21 shows, this factor 
had both main effects and quadratic effects on the models response surface. The 
Prediction Profile for TIGmin reports that as the TIGmin increases there is an 
exponential negative gradient impact on the losses response surface. This may 
be an unanticipated affect that counters the rule constraints associated with the 
enlisted career force controls (ECFC). Specifically, a Marine Agent can not be 
forced to exit the Marine Corps due to being passed over for promotion too many 
times if they are never eligible for a promotion in the first place. 

The second evaluation metric, input significance priority, is observed in the 
Sorted Parameter Estimates report. This report captured 15 terms that varied 
from main effects, quadratic effects, and two-way interactions. The column 
labeled “t ratio” was the measurement metric used to evaluate input significance. 
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Table 3, located at the end of this section, aggregates and compares the terms 
from all response surfaces. 


'' Response Losses 
^ Effect Summary 


Source 

LogWorth 








PValue 

L.EAS 

217.967 



0.00000 
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0.00000 
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^ Actual by Predicted Plot 



Losses Predicted P <.0001 RSq=0.99 RMSE=207.71 

Figure 21. Losses Model Regression Analysis 
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A Summary of Fit 

RSquare 0.988499 

RSquareAdj 0.98778 

Root Mean Square Error 207.7148 

Mean of Response 29403.93 

Obsenrations (or Sum Wgts) 256 

A Analysis of Variance 


Sum of 


Source DF Squares Mean Square F Ratio 

Model 15 889959368 59330625 1375.131 

Error 240 10354908 4314545 Prob > F 

C. Total 255 900314276 <.0001* 


A Parameter Estimates 


Term 

Estimate 

Std Error 

t Ratio 

A 

0 

w 

a. 

Intercept 

22505438 

1 54.9925 

145.20 

<.0001* 

TISmin 

-33.83698 

8.67502 

-3.90 

0.0001* 

(TISmin-0.49219)*(TISmin-0.49219) 

-22.74498 

6.166894 

-3.69 

0.0003* 

TIGmin 

-237.9692 

10.56859 

-22.52 

<.0001* 

(TIGmin-0.99609)*(TIGmin-0.99609) 

-168.1322 

9.050977 

-18.58 

<.0001* 

L.Boot 

746.37632 

89.83388 

8.31 

<.0001* 

L.NEAS 

2520.6538 

89.43311 

28.18 

<.0001* 

LEAS 

11002848 

89.53303 

122.89 

<.0001* 

(L.EAS-0.8496)*(L.EAS-08496) 

-1060146 

736.1404 

-1440 

<.0001* 

L.Reup 

-5233.99 

89.68576 

-58.36 

<.0001* 

(TISmin-0.49219)*(L.Boot-085081) 

7.8013236 

64.07839 

0.12 

0.9032 

(TISmin-0.49219)*(L.EAS-08496) 

-227.1354 

54.84371 

-4.14 

<.0001* 

(TIGmin-0.99609)*{L.NEAS-08501 1) 

-395.3276 

76.13827 

-5.19 

<.0001* 

(TIGmin-0.99609)*(L.EAS-08496) 

-227.2533 

82.80352 

-2.74 

0.0065* 

(L.NEAS-0.85011)*(L.EAS08496) 

-4298832 

557.4738 

-7.71 

<.0001* 

(L.EAS-0.8496)'(L.Reup-084937) 

-2305.521 

666.4464 

-3.46 

0.0006* 


A Sorted Parameter Estimates 
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Std Error 

t Ratio 



Prob>|t| 

LEAS 

11002848 

89.53303 

122.89 
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Figure 21 Continued. 


Losses Model Regression Analysis 
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Gains Metamodel 


The second metamodel evaluated was gains. The response surface 
metric was constructed by aggregating all the gain metrics output by the MSM. 
Specifically, this included the gain sub-categories: EAD, Occ, Rever, Cent, 
Broken, Deserter, and other. Reference Appendix C for more information 
regarding MSM’s output metrics. 

Figure 22 shows the seven JMP statistical reports for this metamodel. This 
model had an excellent fit, indicated by its 0.95829 adjusted R-Square value. 
This is visually reinforced with the “Actual by Predicted Plot,” where the model’s 
response surface equation predicted a forecast that was within 4.2% of the actual 
simulated output measurements. 

For our first metric, the metamodel yielded excellent correlation with 
respect to the gain factors. Four out of the seven gain factors were identified as 
being significant. Additionally, the four selected gain factors all had positive 
gradients, which coincides with the FIRDP. Conversely, for factors of losses were 
determined to be significant, where two had positive gradients and two had 
negative gradients. We viewed this as being positive, because even though they 
were determined to be significant their gradient directions offset each other. 
Therefore there aggregated effect had little to no impact on the metamodel. Two 
factors that were a surprised to have been identified as significant were TISmin 
and TIGmin. 

An important item to reiterate is that the Gains process is separate from 
the Accessions process. Where the later deals predominately with new recruits 
entering the Marine Corps, the Gains process focuses on Marines with prior 
service that re-enter the Marine Corps’ manpower accountability. 
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'' Response Gains 
Effect Summary 


Source 

LogWorth 

G.Occ 

116.149 

G.Dstr 

108.249 

G.Oth 

74.210 

TIGmin 

48.179 

TISmin 

28.073 

L.EAS 

20.575 

G.EAD 

19.262 

TIGmin*TlGmin 

17.492 

TISmin*TlGmin 

12.079 

TlSmin*nSmin 

10.596 

TIGmin’G.Dstr 

4.161 

L.Reup 

3.904 

L.NEAS 

2.587 

L.Boot’G.EAD 

1.080 

L.NEAS'L.Reup 

0.766 

G.Dstr'G.Occ 

0.441 

TISmin'LEAS 

0.302 

L.Boot 

0.209 

TIGmin'G.Oth 

0.069 

Remove Add 

Edit □ FDR 


PValue 

0.00000 

0.00000 

0.00000 

0.00000 

0.00000 

0.00000 

0.00000 

0.00000 

0.00000 

0.00000 

0.00007 

0.00012 

0.00259 

0.08313 

0.17146 

0.36206 

0.49940 

0.61867 

0.85242 


A 


^ Actual by Predicted Piot 



Gains Predicted P<.0001 RSq=0.96 RMSE=15.722 


A Summary of Fit 

RSquare 

0.958291 

RSquare Adj 

0.954933 

Root Mean Square Error 

15.72186 

Mean of Response 

731.3281 

Observations (or Sum Wgts) 

256 

A Analysis of Variance 

Sum of 


Source DF Squares 

Mean Square F Ratio 


Model 19 1340262.7 70540.1 285.3831 

Error 236 583338 2472 Prob > F 

C. Total 255 1 3985964 <.0001* 

Figure 22. Gains Model Regression Analysis 
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A Parameter Estimates 


Term 

Estimate 

Std Error 

t Ratio 

A 

A 

O 

Intercept 

-65.55487 

16.38959 

-4.00 

<.0001* 

TISmin 

8.4038675 

0.657177 

12.79 

<.0001* 

TIGmin 

15.088175 

0.801316 

18.83 

<.0001* 

L.Boot 

-3.389651 

6.801107 

-0.50 

0.6187 

LNEAS 

20.61325 

6.769176 

3.05 

0.0026* 

LEAS 

70.920336 

6.776908 

10.46 

<.0001* 

L.Reup 

-26.49265 

6.790213 

-3.90 

0.0001* 

G.EAD 

68.393713 

6.810322 

10.04 

<.0001* 

G.Dstr 

276.51739 

6.770917 

4084 

<.0001* 

G.Oth 

184.89204 

6.77649 

2728 

<.0001* 

G.Occ 

303.42149 

6.810352 

44.55 

<.0001* 

(TISmin-0.49219)*(TIGmin-0.99609) 

-4.247221 

0.561019 

-7.57 

<.0001* 

(TISmin-0.49219)*(L.EAS-08496) 

2.7941542 

4.130475 

0.68 

0.4994 

(nGmin-0.99609)*(G.D5tr-084986) 

24.180282 

5.967729 

4.05 

<.0001* 

(TIGmin-0.99609)*(G.Oth-085012) 

-0.976455 

5.243185 

-0.19 

0.8524 

(L.Boot-0.85081)*(G.EAD4)85087) 

-88.10257 

50.62888 

-1.74 

0.0831 

(L.NEAS-0.85011)*(L.Reup4)84937) 

65.042312 

47.41715 

1.37 

0.1715 

(G.Dstr-0.84986)*(G.Occ4)34909) 

47.223294 

51.71095 

0.91 

0.3621 

(nSmin-0.49219)*rriSmin-0.49219) 

3.2598 

0.465246 

7.01 

<.0001* 

(TIGmin-0.99609)*(TIGmin-0.99609) 
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Sorted Parameter Estimates 

Term 

Estimate 

Std Error 

t Ratio 



G.Occ 

303.42149 
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Figure 22 Continued. Gains Model Regression Analysis 
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3. 


Promotions Metamodel 


Figure 23 depicts Promotions metamodel. For the response surface, we 
used the direct output metric promotions in (Promln). The Actual by Predicted 
Plot report showed signs of banding within the data set. Banding is a term that 
refers to output measures that do not have a continuous distribution. Instead, the 
data groups together in clusters. This implies the model hosted an aspect of 
discrete outcomes. To capture these discrete bands, we used JMP’s functionality 
to create a partition tree on the data set. As anticipated, the first split identified 
the factor TIGmin as having a discrete impact on the model. A split categorized 
the factor into leaves <3 and >3, and was responsible for explaining 52% of the 
original variance in the model. 

With the Partition Tree’s leaves established, a standard least squares 
regression was re-run, this time incorporating the leaves into its regression. The 
recalculated regression showed a much better model fit using the hybrid 
regression. The adjusted R-Square value increased from 81% to 94%, justifying 
that this was the correct model to explain the Promotions response surface. 

Focusing on the Prediction Profile report, this model showed a lot of 
expected behaviors. For the loss rates of retirement, NBAS, and EAS, all three 
had positive gradients. This fact supports the billet availability FIRDP constraint in 
that availability must exist in order for a Marine to be promoted into a higher 
grade. These loss rates confirm that as the loss rate increases, that is more 
Marines are exiting the Corps, the more billets become available for junior 
Marines to promote into. Additionally, the factor TISmin yielded a profile that 
supported the FIRDP. Specifically, if the ECFC constraints require a greater 
TISmin for a particular billet, that ECFC is therefore decreasing the pool of 
possible candidates that could be considered for promotion. 
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^ '' Response Promotions 
^ Effect Summary 


Source 

LogWorth 

Leaf Label 

125.766 i 
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5.0951 

L.EAS*L.EAS 
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Remove Add Edit I I FDR 
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0.14702 
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.4 Summary of Fit 

RSquare 0.945581 

RSquareAdj 0.943127 

Root Mean Square Error 301 8.32 

Mean of Response 65887.09 

Observations (or Sum Wgts) 256 

Analysis of Variance 

Sum of 

Source DF Squares Mean Square F Ratio 

Model 11 3.8625er10 3.5113er9 385.4272 

Error 244 2222902164 91102548 Prob > F 

C. Total 255 4.0848e-10 <.0001* 


Figure 23. Promotions Model Regression Analysis 
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^ Parameter Estimates 


Term 

Estimate 

Std Error 

t Ratio 

Prob>|t| 

Intercept 

25178424 

2247.606 

11.20 

<.0001* 

Leaf Label[TIGmin<3] 
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48.10 
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L.NEAS 

6875.724 

1299.715 

5.29 
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L.Reup 

-22296.19 

1303.962 

-17.10 
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-4.56 

<.0001* 

(L.Ret-0.85035)*(L.NEAS-085011) 

-1019343 
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-0.83 

0.4056 
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<.0001* 

(L.EAS-0.8496)*(L.EAS-08496) 

-3647929 

10579.58 

-3.45 

0.0007* 

A Sorted Parameter Estimates 
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Figure 23 Continued. Promotions Model Regression Analysis 

4. LatMoves Metamodel 


Figure 24 depicts lateral moves (LatMoves) metamodel. For the response 
surface, we used the direct output metric lateral moves in (LMIn). The LatMoves 
model returned the second best Summary of Fit report characteristics when 
compared to the other metamodels. Specifically, it calculated an impressive 
adjusted RSquare value of 99%, meaning an almost exact fit between the 
metamodel and the existing data. 

With respect to correlation and gradients, the LatMoves metamodel 
closely mimicked the Promotions metamodel. This makes sense because while 
promotions analysis is concerned with the vertical accession through the MOS- 
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Grade state graph network, LatMoves is concerned with the horizontal 
transitions. In both cases, vacancies must exist to facilitate the need for a Marine 
to LatMove into that identified MOS. 

With respect to term significance, loss rates once again dominated the 
metamodel. This again reinforces the fact that the MSM is built around the billet 
vacancy constraint; that vacancies must exist in order to allow Marine Agents to 
move around within the MOS-Grade state graph network. 


^ ' Response LatMoves 
^ Effect Summary 
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Figure 24. LatMoves Model Regression Analysis 
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I Analysis of Variance 



Sum of 



Source 

DF 

Squares 

Mean Square 

F Ratio 

Model 

18 

2231871.1 

123993 

15 27.231 

Error 

237 

19241.6 

81 

Prob > F 

C Total 

255 

2251112.7 


<.0001* 

1 Parameter Estimates 
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Estimate 

Std Error t Ratio Prob>|t| 


Intercept 
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21.53 
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TlGmin 
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3.900388 

-4.71 

<.0001* 

(TIGmin-0.99609)*(L.Ret-085035) 

21.866344 

3.392504 

6.45 

<.0001* 

(TIGmin-0.99609)*(L.NEAS-085011) 

13.31528 

3.497996 

3.81 

0.0002* 

(nGmin-0.99609)*(L.EAS-08496) 

17.382537 

3.692894 

4.71 

<.0001* 

(L.Ret-0.85035r(L.NEAS4)85011) 

22.141672 

38.47048 

0.58 

0.5655 

(L.Ret-0.85035)*(L.Reup4)84937) 

-77.11275 

26.60797 

-2.90 

0.0041* 
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Figure 24 Continued. LatMoves Model Regression Analysis 
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5. End Strength Metamodel 

Figure 25 depicts the End Strength metamodel. For the response surface, 
we used the direct output metric “End”; refer to Appendix C to view all MSM 
outputs. The End Strength model yielded the worst Summary of Fit report 
characteristics. Flowever, this was expected. Let us first explain the reports 
produced by JMP and then we will justify why it was expected. 

From the initial standard least squares regression, banding was identified 
which suggested a factor was exhibiting discrete impact properties. This was 
confirmed with a Partition Tree. Specifically, the factor TIGmin was again 
identified in having the same categorical impacts of <3 and >3 leaf values. 
Applying a singular split explained 86% of the variance within the model. 

With the Partition Tree’s leaves established, a standard least squares 
regression was re-run, this time incorporating the leaves into its regression. The 
recalculated regression showed a much better model fit using the hybrid 
regression. The Adjusted RSquared value increased from 72% to 92%, justifying 
that this was the correct metamodel to explain the End Strength response 
surface. 


A ” Response End Strength 
^ Effect Summary 


Source 
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PValue 
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Figure 25. End Strength Model Regression Analysis 
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^ Actual by Predicted Plot 



RMSE=2066 


A Summary of Fit 

RSquare 0.925654 

RSquareAdj 0.922934 

Root Mean Square Error 2066.01 1 

Mean of Response 151813.6 

Observations (or Sum Wgts) 256 


A Analysis of Variance 

Sum of 

Source DF Squares Mean Square F Ratio 


Model 
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Error 
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C. Total 
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Figure 25 Continued. End Strength Model Regression Analysis 
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As previously stated, the model’s poor fit characteristics were expected. 
This is due to the constraint applied by the MSM that vacancies must exist to get 
Marine Agents to move within the model. However, the overall quantity of 
vacancies is capped by the input GAR force structure. For this experiment, the 
same GAR structure progression was used for every DOE design point. This 
ensured that every design point simulation run had a fourth-year GAR end 
strength input value of 154,836 personnel. 

Figure 26 shows the End Strength output measurement’s distribution. 
Specifically it shows a histogram of End Strength values for each of the 
256 design points, as well as a statistical summary for that histogram. The 
produced histogram visually shows that an overwhelmingly majority of the 256 
design points yielded End Strength calculations that were tightly coupled next to 
the GAR’S fourth forecasted year. Secondly, the Summary Statistics report 
shows the statistical mean calculated from this distribution, and the respective 
95% confidence interval that is tightly coupled around the mean. 

To summarize, these facts prove that the MSM was built to align with 
future GARs. Therefore, GARs are without a doubt the most crucial and 
significant input for the simulation. It also yields warning that the data output 
metrics are directly dependent on the GAR. Meaning the manpower forecasts 
produced from adjusting any of the any internal constraints, like ECFCs, are only 
accurate for the specific GAR it ran its simulation off of. 
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Figure 26. DOE End Strength Distribution Analysis 


6. P2T2 Metamodel 

Figure 27 depicts the Patients, Prisoners, Transients, and Trainees (P2T2) 
metamodel. For the response surface, we used the direct output metric “P2T2”; 
refer to Appendix C to view all MSM outputs. Immediate signs with the P2T2 
model suggested that discrete factors were having an impact on its response, so 
a partition tree was used to identify the data bandings that were occurring. Once 
again TIGmin was identified as having a discrete impact on the model. A split 
categorized the factor into leaves <3 and >3, and was responsible for explaining 
86% of the original variance in the model. 

With the Partition Tree’s leaves established, a standard least squares 
regression was re-run, this time incorporating the leaves into its regression. The 
recalculated regression showed a much better model fit using the hybrid 
regression. The adjusted RSquare value increased from 74% to 95%, justifying 
that this was the correct metamodel to explain the Gains response surface. 

It was originally anticipated that the input significance priority would have 
been more equally distributed across a larger variety of factor terms. As 
explained in Chapter III, the P2T2 is calculated from the instantiation of a Marine 
Agent being selected for a LatMove. Therefore, if the quantity of LatMoves 
increases, so should the quantity of P2T2. Some of this does show through in 
Figure 27 with respect to the positive correlation value for EAS and the negative 
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correlation value with Reenlistment (Reup). Specifically, as these two factor 
terms increase in positive and negative values, respectively, more vacancies 
would open up in MOS-Grade states which therefore would give more Marines 
the opportunity to LatMove into a new MOS. 


^ Response PT2P 
^ Effect Summary 


Source 

LogWorth 
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Analysis of Variance 



Source 

Sum of 
DF Squares 

Mean Square 

F Ratio 

Model 

15 3723750856 

248250057 

349.1426 

Error 

240 170646626 

711027.61 

Prob > F 

C Total 

255 3894397482 


S.0001’ 


Figure 27. P2T2 Model Regression Analysis 
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A Parameter Estimates 


Term 

Estimate 

Std Error 

t Ratio 

Prob>|t| 

Intercept 

13219.674 

644.8322 

20.50 

<.0001* 

Leaf Label[TIGmm<3] 

5503.3663 

104.5987 

52.61 

<.0001* 

TISmin 

-254.1741 

35.24504 

-7.21 

<.0001* 

TIGmin 

-43.06767 

55.1642 

-0.78 

0.4357 

PrmTgt 

59.448439 

30.31253 

1.96 

0.0510 

LNEAS 

1018.2579 

363.1178 

2.80 

0.0055* 

LEAS 
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L.Reup 

-3397.674 
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-1755.609 
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209.7159 

-2.94 

0.0036* 
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-7.52 
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2999882 

-3.42 
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A Sorted Parameter Estimates 
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Figure 27 Continued. P2T2 Model Regression Analysis 


7. Retention Metamodel 


Figure 28 depicts Retention metamodel. The response surface metric was 
constructed by aggregating all the “In-Year,” “Out-of-Year,” and “Not Accounted 
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for” subcategories of MSM’s retention metrics. Reference Appendix C for more 
information regarding MSM’s output metrics. 

The Retention model returned the best Summary of Fit characteristics of 
any metamodel. Specifically, it boasted an impressive adjusted RSquare value of 
99.9%. Additionally, the significant factor terms had correlation signs that aligned 
with the HRDP process. Specifically, the EAS loss rate had a predominate effect 
on the response surface of the model, which is expected because if Marines 
reach their EAS and leave the Marine Corps there will be a dramatic drop in the 
candidate population that could reenlist. 


^ '' Response Retain Sum 
A Effect Summary 


Source LogWorth 
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1 Anaiysis of Variance 



Source 

Sum of 
DF Squares 

Mean Square 

F Ratio 

Model 

17 4857003008 

285706059 

14640.66 

Error 

238 464446427 

19514.556 

Prob > F 

C Total 

255 4861647472 


<.ooor 


Figure 28. Reenlistment Model Regression Analysis 
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A Parameter Estimates 


Term 

Estimate 

Std Error 

t Ratio 

A 

0 

1^ 

Intercept 

55366393 

127.9425 

432.74 

<.0001* 

TIGmin 

-92.08955 

7.107783 

-12.96 

<.0001* 

L.Ret 

-987.9805 
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A Prediction Profiler 
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Figure 28 Continued. Reenlistment 


Model Regression Analysis 
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Accessions Metamodel 


Figure 29 depicts Accessions metamodel. For the response surface, we 
used the direct output metric non-prior service (NFS) accessions. Immediate 
signs with the Accessions metamodel suggested that discrete factors were 
having an impact on its response, so a partition tree was used to identify the data 
bandings that were occurring. Once again TIGmin was identified as having a 
discrete impact on the model. A split categorized the factor into leaves <3 and > 
3, and was responsible for explaining 85% of the original variance in the model. 

With the Partition Tree’s leaves established, a standard least squares 
regression was re-run, this time incorporating the leaves into its regression. The 
recalculated regression showed a much better model fit using the hybrid 
regression. The adjusted RSquare value increased from 75% to 95%, justifying 
that this was the correct metamodel to explain the Gains response surface. 

^ •’ Response Accessions 
A Effect Summary 
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Figure 29. Accessions Model Regression Analysis 
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^ Actual by Predicted Plot 
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J Analysis of Variance 


Source 

Sum of 
DF Squares 

Mean Square 

F Ratio 

Model 
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Figure 29 Continued. Accessions Model Regression Analysis 
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Sorted Parameter Estimates 
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Figure 29 Continued. Accessions Model Regression Analysis 

9. Effect Significance 

The purpose of this section was to analyze the significant terms calculated 
by each metamodel. The metric for measuring this was the calculated “t Ratio,” 
which is viewed in the Sorted Parameter Estimates report, “t Ratio” is the ratio of 
the estimate when compared to its standard error (“Estimates,” n.d.. Sorted 
Estimates section). The final product of this analysis was Table 4. 


While it can be positive or negative in order to identify the term’s gradient 
direction, our analysis chose to focus on overall effect significance vice gradient. 
To accomplish this, we took the absolute value of each term’s value, aggregated 
and averaged that term, where blank cells received a value of zero, and then 

sorted the terms in descending order. Blank cells in Table 4 represent instances 
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where that term was not identified as being significant for that particular 
metamodel. Main effects factors are bolded for easy viewing. 


Table 4. Aggregated Effects and their Priority of Significance 


Input 

Losses 

Gains 

Promotions 

LatMoves 

End 

Strength 

P2T2 

Re- 

Enlist 

Accessions 

Absolute 
Value Avg 
t Ratio 

L.EAS 

122.89 

10.46 

36.77 

133.39 

-7.87 

12.66 

-485.7 

15.53 

103.16 

L.Reup 

-58.36 

-3.9 

-17.1 

-84.48 


-9.32 

-78.82 

-10.62 

32.83 

L.NEAS 

28.18 

3.05 

5.29 

34.84 

-1.85 

2.8 

-72.44 

4.78 

19.15 

TIGmin 

-22.52 

18.83 



-0.64 

-0.78 

-12.96 

-0.81 

7.07 

L.EAS'L.EAS 

-14.4 


-3.45 



-3.42 

30.3 

-3.47 

6.88 

L.Ret 



4.98 

28.37 



-16.41 


6.22 

G.Occ 


44.55 


-4.71 





6.16 

TISmin 

-3.9 

12.79 

-9.16 


-4.41 

-7.21 


-7.21 

5.59 

TIGmin'TIGmin 

-18.58 

9.46 


2.93 



-10.56 


5.19 

G.Dstr 


40.84 







5.11 
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-3.69 

7.01 
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-7.49 
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-6.99 

-4.37 
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-4.41 

0.68 
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-6.38 

-7.74 


-7.64 

3.93 
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-9.99 



13.55 


3.56 
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-7.71 






20.42 


3.52 
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3.41 
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-3.16 

-2.74 
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2.96 
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10. Database Pull Inputs 

During the course of this experiment, a pattern emerged with respect to 
the database pulls from Total Force Data Warehouse (TFDW), invoked by 
MSM’s source code, and how they were utilized as inputs for the MSM. While we 
were writing and debugging our test-main Java class, the class that would run 
the DOE experiment, it was observed that the TFDW data pull “Reenlistments” 
had zero effect on any of the output measurements. Examining this further, it was 
determined that while MSM pulls “Reenlistments” data from TFDW and stores it 
as an object within the MSM, the MSM never again references or invokes this 
data. While the level of significance for the other three database pulls was not 
examined nor determined, due to how this experiment was setup and run, the 
after-action knowledge gained for this insignificant impact was worth annotating 
in the analysis of the DOE experiment 


77 



THIS PAGE INTENTIONALLY LEFT BLANK 


78 



V. MSM VALIDATION 


This chapter examines the accuracy of the model with respect to its ability 
to forecast into the future. To complete this evaluation, historical data from the 
past 12 fiscal years was obtained. This data served two key requirements in 
order to complete the validation study. First, a specific fiscal year’s data served 
as the various inputs that are required to initialize the MSM. As discussed in 
Chapter III, this required extracting a specific fiscal year’s GAR force structure, 
initial inventory, all the reference inputs that specify the manpower pyramid 
transition rates, and ECFCs. 

Secondly, the data was used to compare MSM’s output measurements 
against the Marine Corps’ historic manpower quantities. This comparison was 
made at each fiscal year iteration within the MSM. It was decided to have each 
simulation forecast out to five years per simulation run as the majority of enlisted 
contracts conclude by the five-year mark, and therefore would get one complete 
cycle of Marine Agents through the program (Garrick, 2014). As Figure 30 
shows, each simulation run was iterated such that the previous run’s first 
forecasted year became the MSM’s inputs. 

The metric for measurement during this validation was the computed 
relative error for the various categories of each forecasted year. Equation (6) 
depicts the formula for calculating this error were X serves as the measured 
mean for a particular category, and t is its actual historical value (Law & Kelton, 
2000 ). 


r 

Using relative error allowed for the formula to naturally normalize a data 
set to its respective fiscal year target. This was crucial as the numerous 
manpower historic quantities naturally fluctuate year to year. The equation also 
standardized the calculations; negative calculations indicated the MSM 
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underestimated that metric while positive calculations indicated the MSM 
overestimated. 
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Figure 30. Scope of Validation 

It was decided to seed the Mersenne Twister’s Random number 
Generator with the sequence 224. Seeding the random number generator 
ensures that each simulation will follow the exact same random number 
progression. This action removes any dependence on the random number 
generator, and therefore enabled us to compare MSM’s forecasted years per 
simulation run amongst each other. To ensure an appropriate distribution of 
output results, 30 repetitions were conducted per each simulation run. 
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A. 


UNCONTROLLABLE FACTORS 


It is critical to understand that this validation experiment measured how 
the MSM deviates from historical manpower progressions. However, the derived 
accuracy of this validation experiment is based off of the assumption that the 
historical manpower progression is constant, and therefore is controllable. This 
unfortunately does not marry up ideally with Marine Corps’ history. It should 
come to no surprise that numerous uncontrollable factors have occurred 
throughout out the range of the fiscal years that were selected for this validation. 
Events such as Operating Iraqi Freedom (OIF), Operating Enduring Freedom 
(OEF), sequestration, drawing down the Department of Defense’s manpower 
force structure, as well as natural changes in command and leadership all 
influence the direction the Marine Corps intends to go and the manpower 
required to accomplish it. This is being stated because the MSM’s ability to 
accurately forecast is dependent on the future inventory structure GARs that 
serve as its inputs, as well as the transition rates it uses for estimating Marine 
Agent movement. Therefore, while it is reasonable to assume that each 
successive forecasted year is anticipated to experience larger relative error, 
external real-life events have impacted the Marine Corps to alter its previously 
planned manpower force structure. 

B. SOURCE CODE CONSTRAINTS 

During the course of the validation experiment, we identified that the 
retention zones that are hard coded into the MSM’s source code do not align 
exactly with current Marine Corps policy. Specifically, MSM defines these zones 
as: 

• Zone A: less than 5 years of service 

• Zone B: less than 9 years of service. 

• Zone C: less than 13 years of service. 

• Zone D: less than 17 years of service. 

• Senior Enlisted Zone: greater than or equal to 20 years of service. 
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Current Marine Corps policy defines these zones as: 

• Zone A: less than 6 years of service 

• Zone B: less than 11 years of service. 

• Zone C: less than 15 years of service. 

• Zone D: less than 19 years of service. 

• Senior Enlisted Zone: greater than or equal to 19 years of service. 

Because we were unable to determine if these were tuning parameters 
that were set by the developer to manually adjust the MSM in hopes of producing 
more accurate results, we decided to leave these as is in order to effectively 
evaluate the MSM program that is currently being used by M&RA. 

C. RESULTS 

Maintaining consistency with Chapter IV’s Design of Experiment, the 
following categories were examined: aggregated losses, aggregated gains, 
promotions to the effect of drilling down into promoting-to grade data fields, end 
strength, and retention to the effect of drilling down into retention zone data 
fields. To normalize how each validation result was graphed, common 
procedures were enacted. Each simulation run was labeled by the first year it 
forecasted. For example if the user loaded 2005’s inventory and transition rate 
data into the MSM, with 2006’s GAR inventory structure and Enlisted Career 
Force Controls, fiscal year 2006 would be the first forecasted year and therefore 
would also be the label name for that particular simulation run. When plotting 
each categories relative error results, it was decided to arrange the data such 
that numerical forecasted years served as the independent variable on the y-axis 
and the relative errors served as the dependent variable on the x-axis. This data 
arrangement facilitated the ability to overlay the simulation runs in order to 
determine common patterns of progression within the MSM. 

Figure 31 through Figure 51 are the relative error graphs for each 
validation category. To reduce the clutter of the graphs, as well as focus on time 
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Relative Error 


periods that are less centered on OIF and OEF, the graphs show only simulation 
runs stating at fiscal year 2009 and beyond. The full relative error results can be 
viewed in matrix form within Appendix G. 
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Points were calculated with Equation 6. X was the sum of the outputs Neas.n 
and Neas.e for the targeted year, rwas the quantity of Marines pulled from 
TFDW with the separation code A/E/ASfor the targeted year. 

Figure 31. NBAS Losses Relative Error 
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Points were calculated with Equation 6. X was the sum of the outputs Retired.n 
and Retired.e for the targeted year, rwas the quantity of Marines pulled from 
TFDW with the separation code Retired tor the targeted year. 

Figure 32. Retired Losses Relative Error 
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Points were calculated with Equation 6. X was the sum of the outputs Other.n 
and Other.e for the targeted year, rwas the quantity of Marines pulled from 
TFDW with the separation code Other tor the targeted year. 

Figure 33. Other Losses Relative Error 
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Points were calculated with Equation 6. X was the sum of the outputs Eas.n 
and Eas.e for the targeted year, t was the quantity of Marines pulled from TFDW 
with the separation code E/AS for the targeted year. 

Figure 34. EAS Losses Relative Error 
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Points were calculated with Equation 6. X was the sum of the outputs Ead.g, 
Occ.g, Rev.g, Cont.g, Broke.g, Des.g, and Other.g for the targeted year, r was 
the quantity of Marines pulled from TFDW with the same gain codes for that year. 

Figure 35. Aggregated Gains Relative Error 
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Points were calculated with Equation 6. X was the output NPS.g for the 
targeted year, rwas the quantity of Marines pulled from TFDW for total 
accessions for the targeted year. 

Figure 36. Accession Gains Relative Error 
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Points were calculated with Equation 6. X was the tier II output Promln for the 
targeted year at the respective grade, rwas the quantity of Marines promoted to 
E-2 for targeted year, pulled from TFDW. 

Figure 37. Promotions to E-2 Relative Error 
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Points were calculated with Equation 6. X was the tier II output Promln for the 
targeted year at the respective grade, rwas the quantity of Marines promoted to 
E-3 for targeted year, pulled from TFDW. 

Figure 38. Promotions to E-3 Relative Error 
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Points were calculated with Equation 6. X was the tier II output Promln for the 
targeted year at the respective grade, r was the quantity of Marines promoted to 
E-4 for targeted year, pulled from TFDW. 

Figure 39. Promotions to E-4 Relative Error 
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Points were calculated with Equation 6. X was the tier II output Promln for the 
targeted year at the respective grade, rwas the quantity of Marines promoted to 
E-5 for targeted year, pulled from TFDW. 

Figure 40. Promotions to E-5 Relative Error 
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Points were calculated with Equation 6. X was the tier II output Promln for the 
targeted year at the respective grade, rwas the quantity of Marines promoted to 
E-6 for targeted year, pulled from TFDW. 

Figure 41. Promotions to E-6 Relative Error 
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Points were calculated with Equation 6. X was the tier II output Promln for the 
targeted year at the respective grade, t was the quantity of Marines promoted to 
E-7 for targeted year, pulled from TFDW. 

Figure 42. Promotions to E-7 Relative Error 
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Points were calculated with Equation 6. X was the tier II output Promln for the 
targeted year at the respective grade, rwas the quantity of Marines promoted to 
E-8 for targeted year, pulled from TFDW. 

Figure 43. Promotions to E-8 Relative Error 
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Points were calculated with Equation 6. X was the tier II output Promln for the 
targeted year at the respective grade, r was the quantity of Marines promoted to 
E-9 for targeted year, pulled from TFDW. 

Figure 44. Promotions to E-9 Relative Error 
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Points were calculated with Equation 6. X was output Endfor the targeted year, 
r was the total quantity of Marines within TFDW for targeted year. 

Figure 45. End Strength Relative Error 
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Points were calculated with Equation 6. X was the sum of the outputs A.i, A.naf, 
and A.o. Twas the quantity of Marines retained within Zone-A, pulled from 
TFDW for the targeted year. 

Figure 46. Retention Zone-A Relative Error 
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Points were calculated with Equation 6. X was the sum of the outputs B.i, B.naf, 
and B.o. rwas the quantity of Marines retained within Zone-B, pulled from 
TFDW for the targeted year. 

Figure 47. Retention Zone-B Relative Error 
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Points were calculated with Equation 6. X was the sum of the outputs C.i, 
C.naf, and C.o. r was the quantity of Marines retained within Zone-C, pulled from 
TFDW for the targeted year. 

Figure 48. Retention Zone-C Relative Error 
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Points were calculated with Equation 6. X was the sum of the outputs D.i, 
D.naf, and D.o. T was the quantity of Marines retained within Zone-D, pulled from 
TFDW for the targeted year. 

Figure 49. Retention Zone-D Relative Error 
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Points were calculated with Equation 6. X was the sum of the outputs E.i, E.naf, 
and E.o. rwas the quantity of Marines retained within Zone-E, pulled from 
TFDW for the targeted year. 

Figure 50. Retention Zone-E Relative Error 
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Points were calculated with Equation 6. X was the sum of the outputs SR.i, 
SR.naf, and SR.o. rwas the quantity of Marines retained within Zone-SR, pulled 
from TFDW for the targeted year. 

Figure 51. Retention Zone-Senior-Enlisted Relative Error 
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D. VALIDATION ANALYSIS 

On average, the simulation runs that began with 2009 and 2010 typically 
had the worst relative errors for each category This consistency through the 
model suggests that an external, uncontrolled factor, affected manpower data 
and therefore affected the MSM’s validation. Losses and Gains showed similar 
relative error characteristics. Surprisingly, both these graphs produced groupings 
of minimal error that clustered in the second to forth forecasted year range, with 
the larger errors occurring immediately in simulation and then toward the end. 
With regard to promotion, the MSM performed well when forecasting promotions 
to the Staff Non-Commissioned Officer (SNCO) ranks. Promotions to Non- 
Commissioned Officer (NCO) ranks were consistently underestimated. 
Interestingly though, this was reciprocated in the retention zone graphs as most 
of the graphs consistently overestimated the quantity of Marines that would 
reenlist. End Strength overwhelmingly returned the lowest relative error out of all 
the categories evaluated. This once again reinforced that the MSM’s most 
important input is the GAR inventory structure it simulates to. 

The most interesting observation was the correlation between reenlistment 
gains and accessions. All previous investigation identified the MSM to implement 
a simple supply and demand philosophy; once a billet becomes vacant then the 
MSM will exhaust all opportunities to move a Marine Agent into the spot by either 
a lateral move, a promotion, or a lateral move and a promotion simultaneously. 
What was interesting to discover is that the MSM is biased in choosing Marine 
Agents to reenlist vice instantiating Marine Agents through the recruit and 
accessions process. This is seen in the Results Section in that all reenlistment 
zones constantly overestimated targeted values, while the accessions process 
continuously underestimated its target values. 
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VI. CONCLUSION 


A. EVALUATION 

This thesis had three main objectives to accomplish. The first was to 
provide a document that explains the architecture and processes of the MSM. 
The second was to examine how the inputs drive the model, and compare to the 
HRDP. And lastly, we had the objective to evaluate the accuracy of the model by 
a validation approach. 

The MSM is a large complex program. It totals just over 60 Java classes 
for its source code, as well as requires eight libraries to function. This program 
has numerous details that an analysist could spend days, if not weeks, diving into 
and investigating. With respect to the objectives of this thesis, we reached 
several important conclusions. 

• The MSM is a simple supply and demand process. 

While numerous stochastic processes and deterministic algorithms are 
included within the MSM to forecast manpower metrics, these all abide by the 
single rule that a billet vacancy must exist for a Marine Agent to move into it. 
Likewise, if billet vacancies do exist, the MSM will exhaust every inventory option 
to fill any empty MOS-Grade billets. 

• The GAR input is the most important input that drives the MSM and 
its manpower forecasts. 

With the MSM a supply and demand process, the most important input 
that all algorithms are run against is the GAR structure. The GAR sets targeted 
billet quantities for each MOS per grade. While the MSM uses stochastic 
processes and deterministic algorithms to simulate the movement of Marine 
Agents within the Marine Corps, the MSM will always try to fill any vacant billet 
seat in order to achieve an inventory that maximizes every available billet seat 
specified by the GAR. Therefore, the MSM could be viewed more as a tool that 
evaluates future GAR drafts. 
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• The MSM offers many options that allow a user to fine tune 
particular parameters for a simulation run. 

At its foundation, the MSM possesses all the necessary capabilities to run 
simulations and provide forecasted manpower metrics. Specifically, it uses 
historical data pulled from its reference inputs to set the likelihood that manpower 
events occur, and guides the simulation to fully populating the GAR structure. 
Sitting programmatically on top of that, the MSM provides numerous fields within 
its “siminputs” Excel input file that allows the user to fine tune simulation 
parameters. It is recommended to only use these tuning parameters when small 
deviations to input files or ad hoc case scenarios need to be simulated. 
Examples of situations range from adjusting the percentages of male and female 
Marines per allotted end strength to adjusting overall allowable end strength. 
Outside of ad hoc case scenarios, the primary effort should be given to adjusting 
the input GAR structure and selecting archival database reference input 
sequences that closely align with the manpower goals toward which M&RA is 
looking to forecast. 

• The MSM was constructed with a bottom-up methodology that 
captures key processes within the HRDP. 

The power of the model rests in its ability to integrate the numerous 
processes of HRDP into a single simulation model and simulate down into the 
details at the Marine MOS-Grade level. Reinforcing this capability is the plethora 
of data attributes which provide the user with output measurements that can be 
viewed at different Tier levels depending on the level of detail the user wants to 
drill into. The Tier I level provides a simple aggregation of the entire Marine 
Corps for a single forecasted year. Tier II breaks that down further into providing 
MOS-Grade details for a single forecasted year, and lastly Tier III breaks the data 
down even further to provide MOS-Grade details by monthly intervals. 

• The MSM uses a modular architecture structure. 

Each process of the HRDP was constructed as its own Java class, and is 
only invoked when that HRDP is required to be executed. This greatly increases 
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the feasibility of editing the MSM because the editor just needs to ensure the 
parameter(s) being passed to and returned from the invoked Java class remain 
aligned. All other edits can be made within a process with little dependencies on 
the remainder of the MSM. Besides easiness in editing, modularity can also 
facilitate specific policies to be simulated and analyzed. For example, if M&RA 
wanted to update policies on how promotion boards are run, the Promotions Java 
class could easily be edited to reflect a drafted policy, which would ultimately 
produce simulation output metrics that would show HRDP whole effects to a 
single policy change. 

• The MSM is a time-step model with the base time increment set at 
one fiscal year. 

The model simulates the HRDP by employing a waterfall technique for all 
the processes, where a bulk ingest queuing methodology is utilized. Here, the 
entire inventory of Marine Agents line up and execute an itemized process, only 
to move onto the next HRDP process after the last Marine Agent finishes the 
previous one. Additionally, each Marine Agent only has one chance at executing 
a certain event within a process, and then that Marine Agent will have to wait a 
full fiscal year simulation cycle to be afforded another chance. 

• MSM employs rudimentary behavior modeling techniques (artificial 
intelligence). 

For Marine Agents navigating their careers within the MOS-Grade state 
graph network, basic deterministic rules dictate the paths that Marine Agents will 
navigate. They are simply either allowed to do something or they are not; a 
binary decision. The only behavior modeling capability that is given to Marine 
Agents is in the form of statistical probability in a Markov Chain application. For 
the two types of non-deterministic processes used, a Mersenne Twister random 
number generator picks the event a Marine Agent will execute, for that respective 
HRDP process, based on the probabilities of the respective events. 

Overall, the MSM is an exceptionally powerful tool that correctly employs 
the numerous processes of the HRDP. While the level of accuracy of its output 
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measures varies depending on which output measurement is being examined, 
overall it provides tangible data points that prior to this model could have only 
been talked about in theory. For that, it is recommended for continued use and 
incorporating into plans for future improvements. 

B. RECOMMENDATIONS 

While the MSM is a power simulation tool, we identified various areas 
were continued work and research could be applied. 

• Continued Validation 

While this thesis examined the MSM’s input sensitivity with respect to 
eight metamodels and validated the MSM against 21 manpower categories, 
plenty of opportunities exist to dive deeper into the output measures and validate 
their results. For example the eight metamodels focused primarily at the whole 
Marine Corps’ aggregated level, whereas the validation also drilled down into the 
Grade metric. This thesis did not continue examining down into the MOS-Grade 
levels. Tiers II and III of the output measures. Additionally, none of the finance 
capabilities of the MSM were examined in this thesis. 

• MSM Parameter Tuning 

The validation results, illustrated in Chapter V, reveal general patterns to 
MSM’s output metrics accuracy with regard to which year in the future it is 
forecasting for. This implies that parameters could be weighted, and therefore 
tuned, in order to drive that output metric closer to a zero percentage relative 
error. 

• Intelligent behavior 

As previously stated, the Marine Agents follow simple decision rules and 
the simulated selection boards are likewise simple. There are a wide range of 
aspects and areas that could be amended with additional behavior 
characteristics that would render the model a more accurate depiction of real-life 
Marine Corps manpower decisions. For example, a reenlistment bonus behavior 
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could be constructed that improves the likelihood of a Marine Agent to reenlist 
based upon the size of the incentive. 

• Add geographical code and/or Unit Identification Code attributes to 
Marine Agents 

Just as the Marine Corps’ Table of Organization allocates billets to units, 
the MSM could add the additional Unit Identification Code (UlC) attribute to each 
Marine Agent in order to track additional metrics. Specifically, the geographical 
area could be mapped to these codes, and therefore a Permanent Change of 
Station Java class could be constructed that forecasts the financial surge 
associated with Marines moving to new commands in new geographical areas. 
Likewise, a UlC attribute could be added to each Marine Agent. This attribute 
could be used to simulate staffing goals for units as well as more accurately map 
the career progression of a Marine Agent traversing the MOS-Grade state graph 
network. 

• Introduce a Graphic User Interface (GUI) 

Numerous steps exist for setting up and running the MSM. From loading 
all the inputs to setting user defined parameters within the “siminputs” input file, 
these steps require an intricate level of knowledge about the MSM in order to be 
done correctly. Converting the user interaction with the MSM into a GUI would 
better facilitate the process for the user to accurately set up and run the model. 

• MSM output data type 

The current version of the MSM writes its data outputs to a text-type 
document. A few modifications to the outputs Java class could transcribe these 
documents into Excel files, such as comma separated files. Additionally, 
modifications to the Java class could automatically produce specifically sought 
after visual graphics. 

• Remove MSM’s retentions over accessions bias 

A negative aspect of the MSM is that it overestimates Marine Agents that 
choose to reenlist. This has consequences that trickle down through the MOS- 

Grade state graph network and directly affects the accessions forecasts. 
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Specifically, because the MSM employs a supply and demand methodology in 
exhausting all opportunities to retain a Marine Agent, any Marine Agents retained 
are ultimately occupying billets that make it more difficult for lower grade Marine 
Agents to promote upwards. This impacts the inventory for each MOS-Grade, 
which ultimately restricts the quantity of accessions the MSM can instantiate to 
backfill billet vacancies that have trickled their way down to the entry level MOS- 
Grades. 
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APPENDIX A. MSM MODEL INPUTS 


Table 5. Main Sheet Inputs 


Parameter 

Description 

TFDW User Name 

User name if pulling from online database 

TFDW Password 

Password if pulling from online database 

ODSE User Name 

User name if pulling from online database 

ODSE Password 

Password if pulling from online database 

Scenario Name 

Optional scenario name for simulation run 

Number of Years 

Number of years to forecast 

Number of 

Simulations 

Number of repetitions to run the stochastic simulation 

Inventory 

Sequence number from which to pull inventory data 

Initial Forecasted 

Year 

The first year that will be forecasted 

Losses 

If blank program pulls local file. 

Enter comma separated list to pull those years from TFDW/ODSE. 

Reenlistments 

If blank program pulls local file. 

Enter comma separated list to pull those years from TFDW/ODSE. 

Gains 

If blank program pulls local file. 

Enter comma separated list to pull those years from TFDW/ODSE. 

Promotions 

If blank program pulls local file. 

Enter comma separated list to pull those years from TFDW/ODSE. 

Report Options 

Establishes how the program will aggregate its results. 

- Year 

- Year and MOS 

- Year and Grade 

- Year, MOS, and Grade 

- Year and Month 

- Year, Month, and Grade 

- Year, Month, MOS, and Grade 


Table 6. Career Force Controls 


Parameter 

Description 

Grade 

Grade (E1-E9). Each subsequent parameter is broken down and 
evaluated per grade. 

Promotion Targets 

Promotion targets for subject grade 

Minimum TIS 

Controls eligibility for promotion 

Maximum TIS 

Controls eligibility for promotion and retention 

Minimum TIG 

Controls eligibility from promotion 
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Table 6 Continued. Career Force Controls 


Parameter 

Description 

Variable In-Zone 
Selection 

Opportunity 

Controls zone size, selection rate, and passover rate. 

Maximum # 
of Passovers 

Controls whether Marine may continue service 

Limit Type if exceed 
maximum passovers 

Indicates type of control to place if passover 
limit is reached 

TIS Limit if exceed 
maximum passovers 

TIS limit adjustment for those 
who've been passed over 


Table 7. MOS Data 


Parameter 

Description 

PMOS 

Digit indicator of primary MOS 

Gender 

Gender indicator, used on MOS restrictions 

Is BMOS 

Indicates a basic/school training MOS 

Time to Train 
(Non LM) 

Number of days spent in training 
pipeline 

Time to Train 
(LM) 

Number of days spent in training 
when lat moving to this MOS 

Lat Move Only 

Indicates if this MOS is a 'Lat Move Only' MOS 

Title 

Name of MOS 

Converting MOSs 

List of MOSs that are obsolete predecessors of PMOS 

Alias 

List of MOSs that PMOS has been known as in the past 

Comparable MOSs 

List of MOSs that can be used in place of subiect MOS 

Feeds From 

List of MOSs that feed into the subject MOS 

Feeds To 

List of MOSs that the subject MOS feeds into 

Min Grade 

Minimum grade for subject MOS 

Max Grade 

Maximum grade for subiect MOS 

TOE (4) 

4-Year contract 

TOE (5) 

5-Year contract 

TOE (6) 

6-Year contract 
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APPENDIX B. GAR INPUT 


The GAR is a user-provided input to the MSM program. Its data is 
responsible for building the MOS-Grade state graph network of the model, which 
serves as the manpower structure that the Marine Agents navigate through 
during the simulation of their career. The below image is a snapshot of this input: 
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0 
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0 

0 
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T2P2 
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0 

0 

0 
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0 
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0 
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0 
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0 
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ASR 
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0 
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0 

0 
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0 
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0 

0 

0 

0 
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2857 

7802 
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NAR BILLETS 

0 

0 

0 

0 

1479 
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7802 

0 

12138 


T2P2 

0 

0 

0 

0 

54 
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0 
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NAR 
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0 

0 

0 

1533 

3216 

7802 

0 

12551 


GAR 

0 

0 

0 

0 

1533 

3216 

7802 

0 
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0313 

ASR 
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TOTAL B-BILLETS 
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6 
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33 
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3 

0 
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8 

15 

52 

96 
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0 

833 


NAR BILLETS 

8 

15 

52 

96 

140 

174 

348 

0 

833 


T2P2 

0 

0 

1 

3 

5 

18 

0 
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27 
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8 

15 

53 

99 

145 
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15 

53 

99 
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348 

0 
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Figure 52. Snapshot of GAR Input. 


103 


THIS PAGE INTENTIONALLY LEFT BLANK 


104 



APPENDIX C. OUTPUT METRICS 


Table 8. MSM Output Metrics 


Category 

Metric 

Definition 


A-Bil 

Pulled from the GAR input, these are primary, direct billets without allocation of overhead, b-billets, training, etc. The "naive" requirement. 

CO 

B-Bil 

Pulled from the GAR input, these are B-Billets allocated to structure. 

o 

c 

P2T2.gar 

Pulled from the GAR input, these are "Overhead" allocated for patients, prisoners, trainees, and transients. 

3 

Q. 

c 

Gar.u 

This is unadjusted "raw" GAR requirement which doesn't account for allocation of those in boot camp or MOS school. Its formula Is: 

CO 

Q) 

Gar.u = A-Bil + B-Bil + P2T2.gar 

3 

Train. Boot 

Proportional allocation to account for Marines In Boot Camp. 

< 

iS 

"co 

Q 

Train.Mos 

Proportional allocation to account for Marine in MOS School 


GAR adjusted for Boot Camp and MOS School allocation. This is the calculation used to build the MOS-Grade state graph network. Its 

'o 

Gar.a 

formula Is: 

CO 


Gar.a = Gar.u + Train.Boot + Train.Mos 

E 

E 

3 

Start 

Starting inventory pulled from TFDW data. Always based on FY Start (1 Oct) 

CO 

FemStart 

Starting female inventory 


Neas.n 

Quantity of Marine Agents selected to exit the Marine Corps with a Non-EAS separation code. 

tf) w 

CO < 

O LU CD 

0 > -O 

£|<S 

>v = C 

Deaths, n 

Quantity of Marine Agents selected to exit the Marine Corps due to death. 

E20.n 

Quantity of Marine Agents selected to receive an enlisted to officer (e2o) commission, and therefore leave the enlisted manpower structure. 

o-bB 

O) •> CO 

0 > ^ 

Retired.n 

Quantity of Marine Agents selected to exit the Marine Corps due to retiring. 

0 -O g. 

O .9 0 

o ° 

° o 

Other.n 

Quantity of Marine Agents selected to exit the Marine Corps due to other means not documented with a specific separation code. 

Eas.n 

Quantity of Marine Agents selected to exit the Marine Corps due to the end of their enlisted contract called End of Active Service (EAS). 

Boot.n 

Quantity of Marine Agents selected to exit the Marine Corps during bootcamp. 
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Table 8 Continued. MSM Output Metrics 


Category 

Metric 

Definition 

w 

Neas.e 

Quantity of Marine Agents selected to exit the Marine Corps with a Non-EAS separation code. 

w . 

O CO QJ 

0) < "O 

Deaths.e 

Quantity of Marine Agents selected to exit the Marine Corps due to death. 

E20.e 

Quantity of Marine Agents selected to receive an enlisted to officer (e2o) commission, and therefore leave the enlisted manpower structure. 

lfl 

Retired.e 

Quantity of Marine Agents selected to exit the Marine Corps due to retiring. 

f ® % 

Ctj Q. 

O "O o 
« 

o 

Other.e 

Quantity of Marine Agents selected to exit the Marine Corps due to other means not documented with a specific separation code. 

Eas.e 

Quantity of Marine Agents selected to exit the Marine Corps due to the end of their enlisted contract called End of Active Service (EAS). 


Boot.e 

Quantity of Marine Agents selected to exit the Marine Corps during bootcamp. 


Ead.g 

Quantity of Marine Agents selected that return from their extended active duty (EAD) recruiting tour. 

"co 

CC 

o 

o 

o 

LL 

Occ.g 

Quantity of Marine Agents selected that return after having become a deserter. 

Rev.g 

Quantity of Marine Agents selected that return to service with a Rever gain code. 

O 

O) 

Cont.g 

Quantity of Marine Agents selected that return to service with a Cent gain code. 

O 

"cO 

o 

c 

'cO 

Brok.g 

Quantity of Marine Agents selected that return from medical holds. 

Des.g 

Quantity of Marine Agents selected that return after having become a deserter. 

o 

Other.g 

Quantity of Marine Agents selected that return to service for other means not captured. 

Accession 

Forecasts 

NPS.g 

Non-prior service accessions. This field is calculated based on vacancies required to fill requirements or is supplied by the user. 

NPSFem.g 

Same as NPS.g, but for females. There is no required female recruiting, so this is an estimate based on data. 


Promln 

Quantity of Marine Agents promoted into category (depending on tier level, could be aggregrate FY, by MOS-Grade, by month...ect). 

CO 

O 

PromOut 

Quantity of Marine Agents promoted out of a categoy (depending on tier level, could be aggregrate FY, by MQS-Grade, by month...ect). 

o 

LMIn 

Quantity of Marine Agents lateral moved into category (depending on tier level, could be aggregrate FY, by MQS-Grade, by month...ect). 

o 

5 

LMOut 

Quantity of Marine Agents lateral moved out of a categoy (depending on tier level, could be aggregrate FY, by MQS-Grade, by month...ect). 

o 

Q. 

c 

CO 

End 

Quantity of Marine Agents in the inventory at the end of the time iteration (could be either by FY or by month) 

FemEnd 

Quantity of female Marine Agents in the inventory at the end of the time iteration (could be either by FY or by month) 

o 

c 

P2T2 

Quantity of Marine Agents for the aggregated time period (could be either by FY or by month, depends on which output tier viewed). 

0) 

o 

BasePay 

Estimated Base Pay for inventory based on pay tables 


CompPay 

Estimated Composite pay based on by-grade composite planning factors. 
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Table 8 Continued. MSM Output Metrics 


Category 

Metric 

Definition 


EligA 

Quantity of Marine Agents eligible for retention in zone A. 


EligB 

Quantity of Marine Agents eligible for retention in zone B. 


EligC 

Quantity of Marine Agents eligible for retention in zone C. 


EligD 

Quantity of Marine Agents eligible for retention in zone D. 


EligE 

Quantity of Marine Agents eligible for retention in zone E. 


EligSR 

Quantity of Marine Agents eligible for retention in zone SR. 


A.i 

Quantity of Zone A Marine Agents who reenlisted "In-Year.” 


B.i 

Quantity of Zone B Marine Agents who reenlisted "In-Year.” 


C.i 

Quantity of Zone C Marine Agents who reenlisted "In-Year.” 

w 

o 

D.i 

Quantity of Zone D Marine Agents who reenlisted "In-Year.” 

o 

E.i 

Quantity of Zone E Marine Agents who reenlisted "In-Year.” 

c 

o 

SR.i 

Quantity of Zone SR Marine Agents who reenlisted "In-Year.” 

B 

o 

A.naf 

Marine Agents that reenlisted into zone A that were "Not Accounted For" (see end note). 

cc 

B.naf 

Marine Agents that reenlisted into zone B that were "Not Accounted For" (see end note). 


C.naf 

Marine Agents that reenlisted into zone C that were "Not Accounted For" (see end note). 


D.naf 

Marine Agents that reenlisted into zone D that were "Not Accounted For" (see end note). 


E.naf 

Marine Agents that reenlisted into zone E that were "Not Accounted For" (see end note). 


SR.naf 

Marine Agents that reenlisted into zone SR that were "Not Accounted For” (see end note). 


A.o 

Quantity of Zone A Marine Agents who reenlisted "Qut-of-Year" (i.e. EAS in 2017 but Marine Agent reenlisted in 2016). 


B.o 

Quantity of Zone B Marine Agents who reenlisted "Qut-of-Year" (i.e. EAS in 2017 but Marine Agent reenlisted in 2016). 


C.o 

Quantity of Zone C Marine Agents who reenlisted "Qut-of-Year" (i.e. EAS in 2017 but Marine Agent reenlisted in 2016). 


D.o 

Quantity of Zone D Marine Agents who reenlisted "Qut-of-Year" (i.e. EAS in 2017 but Marine Agent reenlisted in 2016). 


E.o 

Quantity of Zone E Marine Agents who reenlisted "Qut-of-Year" (i.e. EAS in 2017 but Marine Agent reenlisted in 2016). 


SR.o 

Quantity of Zone SR Marine Agents who reenlisted "Qut-of-Year" (i.e. EAS in 2017 but Marine Agent reenlisted in 2016). 
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Table 8 Continued. MSM Output Metrics 


Category 

Metric 

Definition 

Other Manpower 
Metrics 

Passovers 

Number of times Marine Agents are passed over for promotion, aggregated. 

PromZoneElig 

Number of Marine Agents eligible for promotion and placed in zone, aggregated. 

PromElig 

Number of Marine Agents eligible for promotion, irrespective of whether or not they were placed in zone. 

sumTTP 

The aggregated sum of Time to Promotion. 

sumTIG 

The aggregated sum of Time in Grade. 

sumYOS 

The aggregated sum of Years of Service. 


The .”naf” category is an artifact of archivai bad data. Prior to the construction of the MSM, it was identified by M&RA that there 
was a ieakage within the retention accountabiiity system due to various HRDP processes and how they interfaced together. 
Specificaiiy, there were Marines who were reeniisting that were not being accounted for by either the “in-year” or “out-of-year” 
retention categories. To account for this group of Marines who were actually reenlisting but not being properly accounted for, the 
developer built the “not-accounted-for” retention category within the MSM (R. A. Garrick, personal communication, April 6, 2016). 
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APPENDIX D. DESIGN ORTHOGONALITY 


Table 9. Design Orthogonality of NOLH DOE 



TIS 

mn 

TSma 

X 

TIG 

Mn 

Rom 

Tgt 

VIZ 

min 

Viz 

Sid 

VIZ 

max 

Pas SO 

Re 

Thsh 

Loss 

Boot 

Loss 

Retir 

Loss 

NEAS 

Loss 

Death 

Loss 

e2o 

Loss 

Other 

Loss 

EAS 

Loss 

Reup 

Gain 

BVD 

Gain 

Destr 

Gain 

Other 

Gain 

occ 

Gain 

Cent 

Gain 

Rever 

Gain 

Brkn 

TlSmin 

1 

0024 

-0 022 

-0.016 

-0.012 

0.02 

-0.028 

-0 003 

-0 04 

-0.002 

-0.01 

0 003 

-0.005 

0.006 

-0 026 

0.007 

-0 018 

003 

0004 

0.007 

-0.014 

0.007 

-0.011 

-0 029 

nSmax 

0024 

1 

0 088 

-3E-04 

-0.002 

-8E-04 

0.023 

-0 032 

-0.005 

-0 004 

0.015 

-0 002 

0 009 

0.016 

0 023 

0.013 

0 011 

-0.007 

-0 004 

0.005 

0 012 

0 003 

0.011 

0007 

HGmin 

-0 022 

0 088 

1 

-0028 

-0.015 

-0.015 

-0.045 

0003 

-0.015 

0029 

0 029 

0 002 

0.011 

8E-04 

0 017 

-0 007 

2E-04 

0.024 

-0 011 

0045 

-0 004 

001 

0.031 

002 

RomTgl 

-0016 

-3E-04 

-0 028 

1 

-0001 

-5E-04 

-0.006 

-0 028 

0.006 

0002 

0 002 

-0 002 

-3E-04 

-0 002 

-0 008 

-0 002 

-0 003 

0 009 

-6E-04 

4E-04 

-0 008 

0 003 

-0 006 

-0 006 

VIZ min 

-0012 

-0.002 

-0015 

-0.001 

1 

-0 003 

-0.005 

-0 007 

0.004 

0005 

0 002 

-3E-04 

-0 004 

-0 002 

-0 008 

-0 004 

-0 005 

0.006 

-0 002 

5E-04 

-0 007 

0 003 

-0 005 

-0 005 

VIZStd 

0.02 

-8E-04 

-0015 

-5E-04 

-0.003 

1 

-0.003 

-0 025 

-0.007 

0003 

7E-04 

-7E-04 

-0.003 

-8E-04 

-0 005 

-6E-04 

-0 002 

0.006 

-0 002 

-0 004 

-0 003 

0 003 

-0 003 

-0 002 

VIZ max 

-0.028 

0023 

-0045 

-0 006 

-0.005 

-0.003 

1 

0006 

0 021 

0.007 

0 004 

-5E-04 

-0 007 

-0 004 

-0.007 

-0005 

-0.008 

0.01 

-0 003 

0.001 

-0.01 

0.005 

-0008 

-0.008 

FfesO 

-0.003 

-0032 

0003 

-0 028 

-0.007 

-0.025 

0.006 

1 

-0 031 

0.005 

-0.02 

-0 02 

-0.01 

0 004 

0037 

0009 

0.01 

-0 002 

-0.005 

-0.017 

-0 009 

-0 009 

0.011 

-0.049 

RsTTlSh 

-0 04 

-0.005 

-0015 

0006 

0004 

-0.007 

0.021 

-0.031 

1 

-0.017 

0.003 

0 005 

001 

-0 007 

-0.009 

-0 022 

0.02 

0.01 

-0.012 

-0.011 

0.014 

-0.06 

0 012 

-0.017 

LBoot 

-0.002 

-0.004 

0029 

0 002 

0005 

0.003 

0.007 

0.005 

-0017 

1 

-0 003 

-0 001 

0 009 

0 006 

0.009 

0005 

0006 

-0 009 

0.001 

-0.002 

0009 

-0 007 

0.01 

0.007 

L Retire 

-001 

0.015 

0.029 

0002 

0002 

7E-04 

0004 

-002 

0 003 

-0.003 

1 

-0.003 

0.004 

0002 

0.004 

0.002 

0.002 

-0 004 

IE-03 

-IE-03 

0006 

-0 003 

0003 

0.004 

LNEAS 

0.003 

-0.002 

0002 

-0.002 

-3E-04 

-7E-04 

-5E-04 

-0 02 

0.005 

-0 001 

-0.003 

1 

0.002 

-5E-04 

0.004 

7E-04 

7E-05 

IE-04 

-IE-04 

6E-05 

0002 

-4E-04 

6E-04 

3E-05 

L Death 

-0.005 

0.009 

0011 

-3E-04 

-0 004 

-0.003 

-0.007 

-0.01 

001 

0009 

0.004 

0 002 

1 

-0 006 

-0.012 

-0 006 

-0 009 

0.009 

-7E-05 

6E-04 

-0 012 

0.007 

-0 009 

-0.008 

Le2o 

0006 

0.016 

8E-04 

-0.002 

-0 002 

-8E-04 

-0 004 

0.004 

-0.007 

0006 

0.002 

-5E-04 

-0 006 

1 

-0 004 

-0 002 

-0 004 

0.006 

-6E-04 

2E-04 

-0 003 

0.005 

-0 005 

-0.003 

Laher 

-0 026 

0.023 

0017 

-0 008 

-0 008 

-0.005 

-0.007 

0037 

-0.009 

0009 

0 004 

0 004 

-0.012 

-0 004 

1 

-0.003 

-0 009 

001 

-0 001 

0.002 

-0 013 

0.005 

-0 011 

-0.007 

LB^S 

0007 

0.013 

-0 007 

-0.002 

-0 004 

-6E-04 

-0.005 

0009 

-0.022 

0005 

0.002 

7E-04 

-0.006 

-0 002 

-0.003 

1 

-0 007 

0.004 

-0 001 

0.001 

-0 004 

0.002 

-0 003 

-0.003 

LReup 

-0018 

0 011 

2E-04 

-0 003 

-0 005 

-0 002 

-0 008 

001 

002 

0006 

0 002 

7E-05 

-0 009 

-0.004 

-0 009 

-0.007 

1 

0007 

-0 002 

0 003 

-0 007 

0.006 

-0 008 

-0 004 

GEAD 

0.03 

-0 007 

0024 

0.009 

0006 

0006 

001 

-0.002 

0.01 

-0 009 

-0 004 

IE-04 

0 009 

0006 

001 

0004 

0007 

1 

0001 

9E-04 

0008 

-0.005 

0.01 

0005 

GDestr 

0004 

-0 004 

-0011 

-6E-04 

-0.002 

-0 002 

-0 003 

-0.005 

-0.012 

0001 

IE-03 

-IE-04 

-7E-05 

-6E-04 

-0 001 

-0.001 

-0 002 

0.001 

1 

-0 002 

-3E-04 

0 003 

-7E-04 

-0 002 

GOther 

0007 

0.005 

0045 

4E-04 

5E-04 

-0.004 

0.001 

-0017 

-0.011 

-0 002 

-IE-03 

6E-05 

6E-04 

2E-04 

0 002 

0.001 

0003 

9E-04 

-0 002 

1 

0001 

-2E-04 

0001 

4E-04 

GOCC 

-0014 

0.012 

-0 004 

-0.008 

-0.007 

-0 003 

-0.01 

-0 009 

0.014 

0009 

0 006 

0 002 

-0.012 

-0 003 

-0.013 

-0 004 

-0 007 

0.008 

-3E-04 

0.001 

1 

0.007 

-0.01 

-0 007 

GCont 

0007 

0.003 

0.01 

0.003 

0003 

0.003 

0.005 

-0 009 

-0.06 

-0 007 

-0.003 

-4E-04 

0.007 

0005 

0.005 

0002 

0006 

-0.005 

0003 

-2E-04 

0007 

1 

0005 

0.005 

GRever 

-0011 

0.011 

0031 

-0.006 

-0 005 

-0.003 

-0.008 

0011 

0.012 

0.01 

0.003 

6E-04 

-0.009 

-0 005 

-0.011 

-0 003 

-0 008 

0.01 

-7E-04 

0.001 

-0.01 

0.005 

1 

-0.008 

GBrkn 

-0 029 

0.007 

0.02 

-0.006 

-0 005 

-0 002 

-0.008 

-0 049 

-0.017 

0007 

0.004 

3E-05 

-0.008 

-0 003 

-0.007 

-0.003 

-0 004 

0.005 

-0 002 

4E-04 

-0 007 

0.005 

-0 008 

1 


Within the table, the top three correlations (the three worst orthogonalities) are highlighted in yellow. 
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APPENDIX E. NOLH DESIGN 


Table 10. NOLH Design 


low level 

-2 

■4 

-1 

-2 

-0.1 

-0.05 

-0.05 

-2 

-2 

0.6 

0.6 

0.6 

0.6 

0.6 

0.6 

0.6 

0.6 

0.6 

0.6 

0.6 

0.6 

0.6 

0.6 

0.6 

high level 

3 

4 

3 

4 

0.05 

0.05 

0.1 

2 

2 

1.1 

1.1 

1.1 

1.1 

1.1 

1.1 

1.1 

1.1 

1.1 

1.1 

1.1 

1.1 

1.1 

1.1 

1.1 

decimals 

0 

0 

0 

1 

3 

3 

3 

0 

0 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 


TIS 

TISmax 

TIG 

Prom 




Pass 

Reinlist 

Loss 

Loss 

Loss 

Loss 

Loss 

Loss 

Loss 

Loss 

Gain 

Gain 

Gain 

Gain 

Gain 

Gain 

Gain 

factor name 

min 

PassO max 

min 

Tgt 

VIZmin 

VizStd 

VIZmax 

over 

Thresh 

Boot 

Retired 

NEAS 

Death 

e2o 

Other 

EAS 

Reup 

EAD 

Desert Other 

OCC 

Cent 

Rever 

Broken 


0 

2 

-1 

-0.6 

0.015 

-0.028 

0.023 

2 

1 

0.713 

1.055 

1.043 

0.704 

0.659 

0.914 

0.621 

0.637 

0.926 

1.073 

0.696 

0.709 

1.004 

0.877 

1.065 


-1 

-1 

0 

-1.6 

0.005 

-0.032 

-0.039 

1 

-2 

0.963 

0.772 

0.821 

1.1 

0.772 

0.877 

0.625 

0.666 

0.952 

1.09 

0.666 

0.838 

0.83 

0.844 

0.938 


0 

2 

2 

4 

-0.061 

-0.004 

0.008 

1 

-1 

0.905 

0.877 

1.065 

0.676 

0.668 

0.961 

1.055 

0.913 

0.627 

1.088 

0.881 

0.895 

0.647 

0.776 

0.709 


-1 

-2 

2 

3.8 

-0.035 

-0.018 

-0.044 

1 

1 

0.768 

0.799 

0.789 

0.957 

0.633 

1.098 

0.903 

0.866 

0.668 

0.959 

0.868 

1.073 

0.786 

0.743 

0.834 


-1 

2 

0 

3.6 

-0.096 

-0.011 

0.016 

1 

1 

1 

0.93 

0.887 

0.641 

1.098 

0.819 

0.862 

0.631 

1.01 

1.096 

1.075 

1.018 

0.678 

0.729 

0.655 


-1 

-2 

0 

3.3 

-0.065 

-0.02 

0.005 

2 

-1 

0.52 

0.76 

0.801 

0.998 

1.1 

0.75 

0.864 

0.793 

0.948 

0.916 

0.95 

0.889 

0.674 

0.633 

0.62 


-1 

4 

3 

-0.5 

-0.047 

-0.034 

0.011 

1 

-2 

1.096 

0.954 

1.012 

0.664 

0.971 

1.041 

0.846 

0.918 

0.659 

0.651 

0.76 

0.823 

1.098 

0.705 

0.776 


-2 

-3 

3 

-0.4 

-0.098 

-0.025 

-0.026 

1 

2 

0.77 

0.651 

0.793 

0.897 

1.036 

0.922 

0.973 

1.073 

0.809 

0.6 

0.778 

0.864 

1.034 

0.766 

0.604 


-2 

4 

-1 

-0.6 

-0.064 

-0.017 

-0.032 

1 

2 

1.055 

0.938 

0.875 

0.637 

0.731 

0.604 

0.745 

0.686 

1.071 

0.811 

0.879 

0.614 

1.002 

0.618 

0.668 


-2 

-4 

-1 

-0.2 

-0.091 

-0.004 

0.01 

2 

-2 

0.69 

0.786 

0.643 

0.907 

0.803 

0.77 

0.723 

0.83 

0.963 

0.801 

0.756 

0.788 

0.936 

0.795 

0.825 


-2 

1 

2 

2.5 

-0.021 

0.021 

0.081 

-2 

-1 

0.668 

0.766 

0.69 

1.096 

0.618 

0.827 

0.635 

0.602 

1.03 

0.657 

0.649 

0.782 

0.924 

0.774 

0.684 


0 

-3 

2 

2.9 

0.038 

0.008 

0.077 

-2 

1 

0.887 

0.92 

1.053 

0.606 

0.645 

0.711 

0.684 

0.606 

1.012 

0.637 

0.766 

0.795 

0.995 

0.784 

0.842 


-1 

2 

0 

3 

0.029 

0.007 

0.084 

-2 

1 

0.799 

0.606 

0.641 

1.069 

0.918 

1.09 

0.934 

0.989 

0.614 

0.795 

0.834 

0.655 

0.979 

0.647 

0.799 


-1 

-2 

-1 

3.4 

-0.003 

0.047 

0.092 

-2 

-1 

1.045 

0.87 

1.071 

0.67 

0.938 

0.936 

0.844 

1.051 

0.6 

0.901 

0.782 

0.717 

1 

0.821 

0.786 


-1 

3 

2 

-0.2 

-0.017 

0.033 

0.053 

-1 

-1 

0.774 

0.807 

0.653 

0.965 

1.026 

0.715 

0.731 

0.731 

0.965 

0.895 

1.065 

1.075 

0.649 

0.748 

0.778 


-1 

-3 

2 

-0.8 

-0.002 

0.008 

0.093 

-1 

1 

0.86 

1.002 

0.961 

0.727 

0.944 

0.606 

0.637 

0.825 

0.938 

1.028 

0.911 

1.028 

0.629 

0.639 

0.791 


-1 

2 

0 

0.2 

0.049 

0.019 

0.038 

-1 

2 

0.663 

0.789 

0.784 

0.983 

0.627 

0.957 

0.852 

1.047 

0.741 

1.061 

0.913 

1.014 

0.6 

0.731 

0.764 


-1 

-3 

-1 

-1.5 

0.027 

0.021 

0.052 

-1 

-1 

1.075 

1.014 

0.901 

0.748 

0.723 

0.862 

0.932 

0.987 

0.733 

0.928 

0.975 

0.998 

0.637 

0.702 

0.621 


-1 

4 

1 

1.1 

-0.064 

0.021 

0.097 

-1 

-1 

1.014 

0.752 

0.83 

0.881 

0.735 

1.059 

0.83 

0.809 

0.987 

1.082 

0.864 

0.694 

1.088 

0.897 

0.961 


-2 

-3 

2 

3.5 

-0.073 

0.012 

0.033 

0 

0 

0.745 

0.905 

1.092 

0.721 

0.707 

1.047 

0.608 

0.784 

0.995 

0.92 

0.799 

0.807 

0.955 

0.93 

0.926 


0 

0 

-1 

3.2 

-0.074 

0.045 

0.04 

-2 

1 

1.071 

0.797 

0.811 

1 

1.018 

0.733 

0.93 

0.946 

0.791 

1.047 

0.762 

0.844 

1.096 

0.862 

1.038 


0 

-1 

0 

1.9 

-0.084 

0.013 

0.043 

-1 

-2 

0.696 

1.084 

0.893 

0.61 

0.926 

0.791 

0.916 

1.075 

0.661 

1.041 

0.87 

0.696 

0.901 

0.989 

0.985 


0 

3 

1 

0.7 

-0.055 

0.001 

0.05 

-1 

-2 

1.012 

0.631 

0.836 

1.086 

0.979 

1.065 

0.829 

0.694 

1.077 

0.641 

0.875 

0.969 

0.729 

1.051 

1.077 


-1 

-1 

3 

0.1 

-0.071 

0.04 

0.027 

-2 

0 

0.748 

0.844 

0.965 

0.758 

1.086 

0.92 

0.772 

0.813 

0.928 

0.678 

1.079 

0.891 

0.606 

0.899 

1.043 


-2 

0 

0 

-2 

-0.046 

0.009 

0.034 

-1 

1 

1.028 

0.873 

0.666 

1.084 

0.752 

0.696 

1.028 

0.92 

0.653 

0.682 

0.979 

0.901 

0.856 

0.809 

0.954 


0 

-3 

0 

-1.1 

-0.087 

0.005 

0.078 

-2 

-1 

0.782 

0.916 

0.873 

0.715 

0.87 

0.729 

0.834 

0.936 

0.743 

0.803 

0.971 

1.069 

0.711 

0.944 

1.075 


0 

1 

1 

2.3 

0.006 

-0.002 

0.049 

-1 

-2 

0.827 

1.059 

0.768 

1.034 

0.815 

0.621 

0.95 

0.651 

1.016 

0.729 

0.973 

0.758 

1.036 

1.03 

0.645 
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0 

0 

3 

1.7 

- 0.013 

- 0.042 

0.051 

-1 

0 

0.989 

0.809 

0.92 

0.75 

0.612 

0.83 

0.987 

0.647 

1.059 

0.762 

1.084 

0.776 

1.038 

0.871 

0.639 

2 

0 

1 

3.7 

- 0.014 

- 0.036 

0.068 

-1 

0 

0.661 

0.895 

0.62 

0.993 

0.932 

0.926 

0.663 

0.873 

0.649 

0.807 

0.893 

0.715 

1.018 

1.006 

0.803 

0 

-3 

1 

1.8 

0.045 

- 0.015 

0.088 

-1 

-1 

0.897 

0.657 

1.002 

0.682 

0.899 

0.998 

0.707 

0.854 

0.704 

0.754 

0.873 

0.637 

1.084 

0.911 

0.717 

0 

4 

2 

1 

0.025 

- 0.005 

0.095 

-2 

-1 

0.823 

0.909 

0.694 

1.038 

0.832 

0.807 

1.004 

0.729 

1.049 

1.069 

0.752 

1.071 

0.735 

0.975 

0.817 

2 

-1 

1 

- 1.4 

- 0.013 

- 0.015 

0.074 

-1 

1 

1.008 

0.838 

1.084 

0.602 

0.879 

0.707 

1.012 

0.704 

0.993 

0.903 

0.741 

0.993 

0.676 

1.018 

0.68 

0 

2 

-1 

0.5 

0.004 

- 0.049 

0.059 

-1 

0 

0.754 

1.071 

0.748 

1.02 

0.678 

0.871 

0.786 

1.002 

0.68 

0.844 

0.651 

0.932 

0.657 

0.95 

0.618 

1 

-1 

1 

0.7 

0.007 

- 0.035 

0.048 

-1 

-2 

0.967 

0.62 

0.942 

0.705 

0.807 

1.006 

0.799 

1.018 

0.612 

1.03 

0.829 

0.975 

0.766 

0.979 

0.643 

1 

2 

2 

2.2 

- 0.057 

- 0.026 

0.099 

0 

-1 

0.914 

0.95 

0.649 

0.92 

0.629 

0.959 

1.077 

0.856 

0.979 

0.86 

1.055 

0.649 

0.94 

0.643 

0.932 

0 

-2 

1 

1.5 

- 0.043 

- 0.032 

0.083 

-1 

1 

0.76 

0.68 

1.086 

0.823 

0.836 

1.069 

0.938 

0.633 

0.842 

0.911 

0.93 

0.663 

0.895 

0.604 

0.934 

0 

3 

0 

1.4 

- 0.092 

- 0.033 

0.071 

0 

1 

0.909 

1.039 

0.803 

0.934 

0.955 

0.776 

0.764 

1.08 

0.764 

0.983 

1.094 

0.83 

0.983 

0.791 

0.973 

1 

-1 

0 

3.8 

- 0.044 

- 0.039 

0.031 

-1 

-1 

0.676 

0.666 

0.981 

0.795 

0.975 

0.735 

0.719 

1.022 

0.788 

0.862 

0.957 

0.735 

0.952 

0.834 

1.09 

2 

1 

2 

-1 

- 0.026 

- 0.02 

0.066 

-1 

0 

1.051 

1.004 

0.815 

0.959 

0.961 

0.86 

1.03 

0.664 

0.901 

0.77 

0.631 

0.887 

0.813 

0.739 

1.047 

0 

-4 

1 

0.5 

- 0.085 

- 0.03 

0.036 

-1 

1 

0.856 

0.731 

0.963 

0.659 

0.998 

0.854 

0.885 

0.61 

0.905 

0.663 

0.678 

1.043 

0.768 

0.696 

1.012 

0 

2 

1 

- 0.1 

- 0.038 

- 0.014 

0.087 

-1 

2 

0.864 

1.065 

0.633 

0.87 

0.84 

0.682 

0.659 

0.905 

0.746 

0.735 

0.784 

1.055 

0.788 

0.606 

0.864 

1 

-1 

-1 

- 0.3 

- 0.032 

- 0.041 

0.09 

0 

0 

0.797 

0.821 

1.038 

0.772 

0.719 

0.799 

0.704 

0.973 

0.643 

0.766 

0.801 

1.096 

0.789 

0.811 

0.928 

2 

3 

2 

4 

- 0.001 

0.038 

- 0.006 

-2 

0 

0.807 

0.647 

0.909 

0.875 

0.737 

0.637 

0.666 

1.041 

0.95 

0.733 

0.776 

0.971 

0.957 

0.664 

0.891 

2 

-3 

1 

3.3 

0.001 

0.034 

0.013 

0 

1 

0.938 

1.026 

0.774 

0.668 

0.69 

0.815 

0.614 

0.883 

0.971 

0.75 

0.817 

0.922 

0.905 

0.817 

0.893 

0 

2 

-1 

2.9 

0.018 

0.039 

0.014 

-1 

2 

0.651 

0.649 

1.098 

1.01 

0.985 

0.911 

1.051 

0.758 

0.768 

0.684 

0.668 

0.866 

0.985 

0.836 

1 

1 

-1 

-1 

1.3 

0.038 

0.045 

- 0.045 

-1 

-2 

1.006 

1.1 

0.844 

0.739 

1.075 

0.975 

1.018 

0.762 

0.657 

0.68 

0.709 

0.934 

1.057 

0.745 

0.854 

0 

2 

3 

- 0.6 

0.045 

0.019 

- 0.025 

-1 

-1 

0.682 

0.637 

0.922 

1.053 

0.983 

0.731 

0.741 

0.909 

1.008 

1.01 

1.002 

0.618 

0.686 

0.793 

0.916 

1 

0 

3 

0.6 

0.024 

0.045 

0.013 

-2 

2 

1.088 

0.952 

0.723 

0.719 

0.995 

0.788 

0.805 

0.981 

0.887 

0.975 

1.09 

0.791 

0.772 

0.848 

1.051 

1 

4 

0 

- 1.5 

0.009 

0.009 

- 0.004 

0 

0 

0.743 

0.832 

0.969 

0.991 

0.795 

0.977 

1.061 

0.803 

0.663 

0.918 

1.063 

0.797 

0.801 

0.764 

0.852 

2 

-3 

1 

- 1.6 

- 0.002 

0.018 

- 0.007 

-1 

-1 

0.948 

0.955 

0.766 

0.848 

0.686 

1.075 

0.92 

0.709 

0.745 

0.995 

0.954 

0.733 

0.672 

0.602 

0.989 

0 

2 

3 

2 

- 0.099 

0.048 

- 0.007 

-1 

-1 

1.067 

0.782 

0.93 

0.879 

0.606 

0.875 

0.825 

0.%1 

0.924 

1.067 

0.657 

0.897 

1.006 

0.901 

0.805 

1 

-1 

2 

2.1 

- 0.083 

0.005 

0.007 

0 

2 

0.61 

1.057 

0.75 

0.688 

0.793 

1.073 

0.819 

1.059 

0.83 

1.051 

0.813 

0.858 

0.866 

0.903 

0.743 

1 

4 

0 

2.7 

- 0.071 

0.01 

- 0.042 

0 

1 

0.885 

0.813 

0.903 

0.932 

0.903 

0.616 

1 

0.823 

0.823 

0.854 

0.907 

1.059 

0.977 

1.088 

0.789 

2 

-3 

0 

3.5 

- 0.031 

0.012 

0.006 

-1 

-1 

0.727 

0.881 

0.637 

0.856 

0.936 

0.672 

1.09 

0.715 

0.686 

1.071 

0.7 

0.825 

0.981 

0.928 

0.692 

2 

2 

2 

- 1.8 

- 0.066 

0.016 

0.024 

-1 

0 

0.875 

0.858 

1.061 

0.987 

0.909 

1.043 

0.612 

0.901 

0.94 

0.768 

0.909 

0.817 

0.752 

1.026 

0.606 

1 

-3 

2 

- 0.9 

- 0.036 

0.001 

- 0.035 

-1 

1 

0.704 

0.983 

0.854 

0.694 

1.002 

1.032 

0.827 

0.862 

0.868 

0.696 

0.838 

0.61 

0.842 

1.092 

0.827 

0 

1 

0 

- 0.4 

- 0.087 

0.006 

0.012 

0 

2 

1.047 

0.7 

0.866 

0.955 

0.641 

0.801 

0.86 

0.688 

0.715 

0.643 

0.989 

0.848 

0.797 

0.991 

0.745 

0 

-1 

-1 

0.1 

- 0.09 

0.036 

0.018 

-1 

-1 

0.621 

0.932 

0.819 

0.842 

0.639 

0.719 

0.963 

0.848 

0.666 

0.645 

1.092 

0.704 

0.653 

1.016 

0.75 

1 

1 

2 

3.9 

0.031 

- 0.016 

0.021 

-1 

0 

0.715 

0.913 

1.055 

0.963 

0.811 

0.614 

1.071 

0.889 

0.942 

0.709 

0.895 

1.012 

1.065 

1.079 

1.069 
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0 

-3 

1 

1.3 

0.027 

- 0.018 

- 0.038 

-1 

0 

1.061 

0.801 

0.832 

0.791 

0.758 

0.661 

0.922 

1.026 

1.096 

0.827 

0.942 

1.098 

0.909 

1.049 

0.887 

0 

1 

1 

1.6 

0.034 

- 0.029 

- 0.029 

-2 

1 

0.659 

1.092 

0.924 

0.977 

0.883 

0.991 

0.788 

0.608 

0.756 

0.602 

0.998 

0.928 

1.012 

0.854 

0.948 

0 

0 

0 

1.7 

0.042 

- 0.042 

0.003 

-1 

-1 

1.082 

0.741 

0.651 

0.621 

1.051 

1.08 

0.774 

0.768 

0.846 

0.834 

1.082 

0.92 

1.092 

0.985 

0.94 

0 

1 

3 

0 

0.003 

- 0.046 

0.017 

-2 

-1 

0.641 

1.028 

0.914 

1.014 

1.08 

0.705 

0.893 

0.995 

1.079 

0.973 

0.6 

0.77 

0.725 

0.873 

1.041 

0 

-1 

1 

0.9 

0.043 

- 0.032 

0.002 

-1 

2 

0.965 

0.633 

0.823 

0.633 

0.809 

0.803 

0.944 

1.028 

0.899 

1.084 

0.735 

0.7 

0.627 

1.055 

1.026 

0 

4 

1 

0.6 

- 0.012 

- 0.023 

- 0.049 

0 

0 

0.78 

0.975 

1.024 

1.065 

0.854 

1.036 

0.668 

0.774 

0.674 

0.926 

0.764 

0.643 

0.704 

1.028 

0.979 

-2 

0 

1 

- 1.1 

0.002 

- 0.016 

- 0.027 

-2 

0 

0.895 

0.817 

0.604 

0.838 

0.711 

0.946 

0.616 

0.789 

0.698 

1.075 

0.704 

0.86 

0.782 

1.059 

0.996 

0 

1 

2 

3.3 

- 0.097 

- 0.05 

- 0.014 

0 

-2 

0.916 

1.022 

0.971 

1.039 

0.68 

0.852 

0.94 

1.088 

0.934 

0.957 

1.039 

1.088 

0.926 

0.782 

0.69 

0 

-1 

3 

3.1 

- 0.033 

- 0.039 

- 0.023 

-2 

0 

0.666 

0.602 

0.709 

0.752 

0.616 

0.942 

1.022 

0.899 

0.897 

0.891 

1.008 

0.959 

1.039 

0.719 

0.612 

-2 

2 

1 

3.3 

- 0.04 

- 0.031 

- 0.024 

0 

0 

0.942 

0.86 

1.032 

0.903 

1.053 

0.762 

0.711 

0.69 

0.616 

0.815 

0.932 

1.036 

0.991 

0.723 

0.748 

0 

.4 

0 

3.7 

- 0.043 

- 0.004 

- 0.034 

-1 

0 

0.834 

0.692 

0.67 

0.817 

1.038 

0.655 

0.618 

0.67 

0.784 

0.989 

0.946 

0.961 

0.852 

0.616 

0.829 

-2 

3 

3 

- 0.1 

- 0.05 

- 0.027 

- 0.005 

0 

-2 

0.871 

0.871 

1 

0.989 

0.954 

1 

1.053 

1.082 

0.922 

0.713 

0.795 

0.666 

0.815 

0.614 

0.754 

-1 

.4 

2 

- 1.7 

- 0.027 

- 0.007 

- 0.021 

-1 

2 

0.766 

0.737 

0.756 

0.788 

0.873 

0.858 

1.047 

0.885 

1.08 

0.823 

0.614 

0.698 

0.825 

0.727 

0.641 

-1 

3 

-1 

0.5 

- 0.034 

- 0.041 

0.004 

-2 

2 

1.043 

1.096 

1.041 

0.893 

0.635 

0.805 

0.795 

0.707 

0.696 

0.756 

0.78 

0.711 

0.623 

0.678 

0.602 

-1 

-3 

-1 

- 0.1 

- 0.078 

- 0.043 

- 0.037 

0 

-2 

0.821 

0.707 

0.743 

0.643 

0.788 

0.746 

0.725 

0.68 

0.848 

0.692 

0.602 

0.752 

0.731 

0.762 

0.877 

1 

0 

1 

1 

- 0.025 

0 

0.025 

0 

0 

0.85 

0.85 

0.85 

0.85 

0.85 

0.85 

0.85 

0.85 

0.85 

0.85 

0.85 

0.85 

0.85 

0.85 

0.85 

1 

-3 

1 

0.3 

- 0.044 

- 0.04 

- 0.05 

-2 

1 

0.981 

0.907 

1.069 

1.008 

0.93 

0.903 

0.877 

1.071 

1.082 

1.014 

0.961 

0.979 

0.954 

0.913 

1.004 

2 

1 

0 

- 1.7 

- 0.058 

- 0.003 

- 0.046 

-1 

0 

0.788 

0.729 

0.6 

0.657 

0.844 

1.088 

1.069 

1.032 

1.098 

0.924 

0.811 

0.981 

0.893 

0.84 

0.975 

2 

-1 

3 

0.6 

- 0.094 

- 0.031 

- 0.041 

-2 

-1 

0.998 

0.977 

1.036 

1.028 

0.631 

0.684 

0.817 

0.635 

0.678 

0.979 

1.059 

1.024 

1.082 

0.946 

1.092 

1 

3 

1 

0.2 

- 0.06 

- 0.025 

- 0.032 

-2 

1 

0.631 

0.709 

0.618 

0.623 

0.778 

0.809 

0.705 

0.6 

0.711 

1.092 

1.03 

0.995 

0.877 

0.983 

0.903 

1 

0 

1 

1.6 

- 0.089 

- 0.029 

- 0.013 

-1 

2 

1.057 

1.073 

0.94 

1.061 

0.848 

1.02 

1.026 

1.043 

0.961 

0.635 

0.772 

0.678 

0.67 

0.954 

0.93 

1 

1 

-1 

3 

- 0.079 

- 0.041 

- 0.01 

-2 

-1 

0.68 

0.719 

0.68 

0.746 

0.643 

0.887 

0.909 

0.879 

0.914 

0.655 

0.664 

0.647 

0.61 

0.815 

1.084 

3 

-1 

1 

3.3 

- 0.027 

- 0.002 

- 0.015 

0 

-1 

1.098 

0.965 

0.946 

0.94 

1.028 

0.645 

0.643 

0.77 

0.61 

0.702 

0.856 

0.62 

0.692 

0.948 

0.899 

1 

4 

2 

1.6 

- 0.089 

- 0.024 

- 0.005 

-1 

0 

0.868 

0.834 

0.696 

0.774 

0.887 

0.647 

0.735 

0.684 

0.793 

0.76 

0.676 

0.621 

0.639 

0.987 

0.965 

3 

0 

-1 

0.2 

- 0.008 

- 0.029 

- 0.011 

-2 

1 

0.627 

0.985 

0.871 

0.885 

1.008 

0.772 

0.897 

0.952 

0.93 

0.791 

1.006 

1.065 

0.987 

0.666 

0.651 

1 

3 

0 

- 0.3 

0.044 

- 0.011 

0.015 

0 

-2 

0.899 

0.758 

0.721 

0.625 

1.034 

0.821 

0.879 

1.045 

0.973 

0.688 

0.852 

0.963 

0.891 

0.842 

0.713 

2 

-3 

1 

- 1.8 

- 0.016 

0 

- 0.046 

0 

-2 

0.778 

0.995 

0.995 

0.899 

0.776 

1.049 

0.715 

0.84 

0.836 

0.664 

0.885 

1.094 

0.838 

0.61 

0.731 

3 

3 

3 

- 0.4 

- 0.005 

- 0.017 

- 0.015 

0 

0 

0.881 

0.854 

0.61 

0.782 

0.741 

0.883 

0.801 

0.7 

0.682 

0.793 

0.969 

0.944 

0.959 

0.707 

0.729 

2 

.4 

0 

3.6 

- 0.011 

- 0.03 

0.003 

-1 

1 

0.717 

0.996 

1.079 

0.864 

0.694 

0.766 

0.971 

1.061 

0.998 

0.881 

0.748 

0.68 

0.77 

0.76 

0.661 

3 

2 

0 

3.2 

0.026 

- 0.03 

- 0.048 

0 

-1 

0.924 

0.711 

0.764 

0.77 

0.799 

0.666 

0.889 

0.928 

0.946 

0.879 

0.842 

0.616 

0.793 

0.668 

0.627 

2 

-1 

3 

1.1 

0.032 

- 0.003 

- 0.014 

0 

-1 

0.817 

0.967 

1.077 

1.026 

1.045 

0.948 

0.68 

0.621 

0.829 

1.039 

0.68 

0.829 

0.817 

0.875 

0.647 

1 

2 

2 

3.6 

- 0.01 

- 0.022 

- 0.041 

-1 

2 

1.094 

0.836 

0.713 

0.811 

1.012 

0.916 

0.746 

0.735 

0.821 

0.864 

0.707 

0.774 

0.645 

0.688 

0.6 
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2 

0 

0 

- 0.7 

- 0.085 

0.013 

0.004 

-1 

1 

0.846 

0.774 

1.039 

0.936 

0.934 

0.932 

0.686 

1.024 

1.045 

0.942 

0.672 

1.039 

0.897 

0.78 

0.821 

1 

2 

0 

0.3 

- 0.03 

0.022 

- 0.028 

-1 

-1 

0.647 

0.979 

0.795 

0.786 

0.862 

1.014 

0.657 

1.008 

1.024 

0.858 

0.803 

0.95 

0.944 

0.62 

0.944 

2 

-2 

1 

1 

- 0.072 

0.046 

- 0.031 

-1 

-1 

0.979 

0.739 

1.022 

1.051 

0.754 

0.836 

0.907 

0.717 

0.731 

1.094 

0.627 

1.049 

1.01 

0.631 

0.993 

2 

2 

2 

0 

- 0.062 

0.023 

- 0.017 

-2 

0 

0.698 

1.09 

0.672 

0.696 

0.713 

0.676 

0.957 

0.745 

0.817 

1.006 

0.62 

1.047 

1.086 

0.768 

1.036 

1 

0 

0 

2.8 

- 0.069 

0.043 

- 0.048 

-2 

1 

1.086 

0.702 

1.045 

1.016 

0.637 

0.952 

0.641 

1.055 

0.895 

0.707 

1.026 

0.754 

0.633 

0.786 

0.918 

1 

1 

1 

2.8 

- 0.086 

0.036 

- 0.03 

-1 

-1 

0.729 

1.079 

0.625 

0.608 

0.858 

0.%3 

0.891 

0.979 

0.977 

0.737 

1.061 

0.608 

0.727 

0.807 

0.913 

1 

-2 

1 

1.2 

- 0.028 

0.002 

- 0.017 

-1 

-1 

0.991 

0.653 

1.014 

0.922 

0.952 

0.608 

1.049 

0.661 

0.672 

0.871 

0.963 

0.692 

0.737 

0.637 

0.959 

2 

1 

2 

2.3 

- 0.061 

0.043 

- 0.022 

-1 

0 

0.858 

0.848 

0.733 

0.729 

0.973 

0.76 

1.1 

0.614 

0.631 

0.698 

0.926 

0.762 

0.778 

0.832 

0.87 

2 

0 

0 

- 1.4 

0.018 

0.011 

0.02 

-2 

2 

0.789 

0.684 

0.954 

1.073 

0.895 

0.717 

0.717 

0.94 

0.866 

0.7 

0.846 

0.885 

1.041 

1.024 

0.614 

1 

2 

-1 

0.8 

- 0.009 

0.046 

- 0.003 

-2 

-1 

0.993 

0.973 

0.858 

0.62 

0.95 

0.844 

0.842 

0.903 

1.036 

0.883 

0.663 

0.868 

1.08 

0.83 

0.723 

3 

-2 

1 

- 0.9 

- 0.024 

0.006 

0.009 

-1 

-1 

0.764 

0.825 

0.955 

0.871 

0.823 

0.889 

1.036 

0.723 

0.639 

0.746 

0.623 

0.957 

0.873 

1.01 

0.672 

2 

3 

2 

- 0.5 

0 

0.013 

- 0.004 

0 

0 

0.838 

0.889 

0.817 

0.698 

0.61 

0.979 

0.948 

0.819 

0.645 

0.704 

0.809 

1.026 

1.016 

1.075 

0.633 

2 

-3 

-1 

2.8 

0.02 

0.009 

- 0.025 

-1 

0 

0.725 

0.713 

0.985 

0.911 

0.676 

0.698 

0.832 

0.836 

1.063 

0.969 

0.877 

0.746 

0.762 

0.967 

0.719 

2 

2 

1 

3.4 

0.02 

0.034 

- 0.013 

0 

-2 

1.022 

0.885 

0.702 

0.731 

0.78 

0.907 

0.655 

0.934 

1.075 

0.868 

0.84 

0.784 

0.733 

1.077 

0.694 

1 

-3 

2 

1.1 

- 0.021 

0.038 

- 0.001 

-2 

-1 

0.756 

0.803 

0.895 

1.071 

1.039 

1.008 

1.024 

0.725 

0.782 

0.922 

0.987 

0.76 

0.641 

0.996 

0.682 

2 

0 

2 

2.5 

0.008 

0.01 

0.019 

-2 

1 

0.889 

1.03 

0.782 

0.651 

0.991 

1.067 

0.887 

0.643 

0.7 

1.032 

0.903 

0.819 

0.631 

0.942 

0.78 

1 

-1 

0 

0.8 

- 0.046 

- 0.05 

0.085 

-1 

1 

1.03 

1.086 

0.809 

0.887 

1.03 

1.01 

1.006 

0.75 

1.092 

1.024 

0.995 

0.745 

0.998 

1.043 

0.666 

1 

0 

0 

- 0.1 

- 0.078 

- 0.047 

0.03 

0 

-2 

0.793 

0.618 

1.018 

0.725 

0.911 

0.987 

1.073 

0.623 

0.885 

1.077 

0.985 

0.854 

0.881 

0.963 

0.856 

2 

-3 

2 

0.4 

- 0.081 

- 0.044 

0.072 

0 

0 

0.995 

1.012 

0.692 

1.055 

0.825 

0.678 

0.604 

0.922 

0.827 

0.977 

0.967 

0.684 

0.916 

1.071 

0.676 

2 

2 

1 

- 0.2 

- 0.067 

- 0.038 

0.062 

-2 

0 

0.674 

0.625 

0.993 

0.805 

0.829 

0.727 

0.606 

0.%3 

0.717 

1.079 

1.067 

0.748 

0.946 

0.887 

0.67 

1 

-1 

1 

3 

- 0.098 

- 0.025 

0.069 

-1 

0 

1.077 

0.946 

0.627 

0.873 

0.866 

1.071 

1.002 

0.741 

0.981 

0.739 

0.653 

1.032 

0.668 

1.008 

0.758 

1 

2 

0 

2.5 

- 0.08 

- 0.023 

0.086 

0 

0 

0.616 

0.676 

0.983 

0.678 

0.819 

1.026 

0.979 

0.743 

0.881 

0.719 

0.643 

1.03 

0.75 

1.065 

0.862 

1 

0 

1 

2 

- 0.067 

- 0.027 

0.028 

-1 

-1 

1.036 

0.944 

0.704 

0.938 

1.004 

0.67 

0.758 

1.084 

0.838 

0.62 

0.612 

0.899 

0.864 

0.895 

0.811 

1 

1 

2 

1.2 

- 0.072 

- 0.02 

0.061 

-1 

0 

0.723 

0.668 

0.852 

0.733 

1.043 

0.618 

0.709 

0.944 

0.647 

0.764 

0.655 

1.077 

0.707 

1 

0.702 

2 

-2 

-1 

0.2 

- 0.02 

- 0.024 

0.068 

-1 

1 

0.75 

0.924 

0.788 

0.924 

1.047 

0.782 

0.918 

0.746 

0.903 

0.852 

1.096 

0.723 

0.84 

0.698 

0.881 

2 

3 

0 

- 1.1 

0.003 

- 0.007 

0.041 

0 

-2 

0.852 

0.778 

0.989 

0.756 

1.016 

0.745 

1.08 

0.829 

1.094 

0.825 

1.016 

0.821 

0.879 

0.659 

0.86 

2 


2 

- 1.2 

- 0.009 

- 0.048 

0.026 

-2 

-2 

0.772 

1.006 

0.752 

0.891 

0.7 

0.944 

0.733 

0.868 

0.825 

0.715 

1.014 

0.727 

1.079 

0.655 

1.063 

3 

3 

2 

- 0.7 

0.004 

- 0.027 

0.05 

-1 

1 

0.891 

0.612 

0.838 

0.612 

0.704 

0.832 

0.754 

0.924 

0.789 

0.647 

1.01 

0.764 

0.868 

0.741 

1.014 

2 

-3 

0 

3.9 

0.025 

- 0.014 

0.07 

0 

1 

0.746 

1.077 

0.684 

1.082 

0.743 

0.866 

1.098 

0.752 

0.86 

0.948 

0.659 

0.914 

0.811 

0.661 

0.967 

2 

2 

0 

3.2 

0.013 

- 0.049 

0.07 

-2 

-1 

1.065 

0.764 

0.975 

0.86 

0.651 

0.795 

0.924 

0.786 

0.889 

0.914 

0.647 

0.873 

0.625 

0.6 

1.002 

2 

-1 

2 

2.7 

0.001 

- 0.026 

0.029 

0 

-1 

0.87 

1.038 

0.727 

0.866 

1.077 

0.996 

0.633 

1.004 

0.635 

0.887 

0.745 

0.911 

0.829 

0.663 

0.905 

1 

3 

2 

2.9 

- 0.019 

- 0.044 

0.058 

0 

1 

0.969 

0.659 

0.84 

0.784 

1.079 

1.051 

0.789 

0.967 

0.844 

0.83 

0.786 

1.067 

0.758 

0.627 

1.022 
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3 

-2 

0 

- 0.4 

- 0.1 

0.014 

0.044 

0 

2 

1.063 

0.664 

0.741 

1.047 

0.967 

1.061 

0.661 

0.805 

0.92 

0.952 

0.719 

0.69 

1.045 

0.819 

0.737 

2 

4 

-1 

0.6 

- 0.096 

0.035 

0.058 

-2 

0 

0.6 

0.957 

0.928 

0.854 

1.096 

1.077 

0.692 

0.604 

0.975 

1.026 

0.723 

0.793 

1.02 

0.686 

0.629 

3 

-1 

2 

- 1.9 

- 0.091 

0.037 

0.094 

-1 

-2 

1.016 

0.69 

0.762 

0.901 

0.786 

0.643 

0.961 

0.998 

0.633 

1.034 

0.682 

0.659 

0.961 

0.653 

0.705 

1 

4 

3 

- 0.6 

- 0.082 

0.028 

0.06 

-1 

1 

0.608 

1.018 

1.094 

0.655 

0.784 

0.6 

0.952 

0.893 

0.807 

0.913 

0.688 

0.6 

0.854 

0.68 

0.815 

2 

.4 

0 

1.9 

- 0.063 

0.048 

0.089 

0 

0 

0.959 

0.639 

0.674 

0.946 

0.608 

1.012 

0.702 

0.663 

0.991 

0.745 

0.983 

0.987 

0.78 

0.735 

0.793 

3 

2 

1 

3.9 

- 0.06 

0.037 

0.079 

-1 

-1 

0.739 

1.045 

1.088 

0.832 

0.76 

1.038 

0.69 

0.842 

1.006 

0.614 

1.071 

1.061 

0.651 

0.778 

0.832 

1 

-2 

3 

2.6 

- 0.065 

0.028 

0.027 

-2 

-1 

0.987 

0.645 

0.657 

0.996 

1.041 

0.786 

1.079 

1.063 

0.774 

0.627 

1.004 

0.991 

0.696 

0.823 

0.635 

2 

1 

2 

3.6 

- 0.055 

0.032 

0.089 

-1 

2 

0.737 

0.928 

0.879 

0.6 

0.928 

0.823 

1.075 

1.034 

0.748 

0.61 

1.034 

0.862 

0.87 

0.856 

0.762 

1 

-2 

0 

-2 

0.011 

0.004 

0.042 

-1 

1 

0.795 

0.823 

0.635 

1.024 

1.032 

0.739 

0.645 

0.788 

1.073 

0.612 

0.819 

0.805 

1.053 

0.924 

0.991 

2 

2 

0 

- 1.8 

- 0.015 

0.018 

0.094 

-1 

-1 

0.932 

0.901 

0.911 

0.743 

1.067 

0.602 

0.797 

0.834 

1.032 

0.741 

0.832 

0.627 

0.914 

0.957 

0.866 

2 

-2 

2 

- 1.6 

0.046 

0.011 

0.034 

-1 

-1 

0.7 

0.77 

0.813 

1.059 

0.602 

0.881 

0.838 

1.069 

0.69 

0.604 

0.625 

0.682 

1.022 

0.971 

1.045 

2 

2 

2 

- 1.3 

0.015 

0.02 

0.045 

-2 

1 

1.08 

0.94 

0.899 

0.702 

0.6 

0.95 

0.836 

0.907 

0.752 

0.784 

0.75 

0.811 

1.026 

1.067 

1.08 

2 

.4 

-1 

2.5 

- 0.003 

0.034 

0.039 

-1 

2 

0.604 

0.746 

0.688 

1.036 

0.729 

0.659 

0.854 

0.782 

1.041 

1.049 

0.94 

0.877 

0.602 

0.995 

0.924 

3 

3 

-1 

2.4 

0.048 

0.025 

0.076 

-1 

-2 

0.93 

1.049 

0.907 

0.803 

0.664 

0.778 

0.727 

0.627 

0.891 

1.1 

0.922 

0.836 

0.666 

0.934 

1.096 

3 

-4 

3 

2.6 

0.014 

0.017 

0.082 

-1 

-2 

0.645 

0.762 

0.825 

1.063 

0.969 

1.096 

0.955 

1.014 

0.629 

0.889 

0.821 

1.086 

0.698 

1.082 

1.032 

3 

4 

3 

2.2 

0.041 

0.004 

0.04 

-2 

2 

1.01 

0.914 

1.057 

0.793 

0.897 

0.93 

0.977 

0.87 

0.737 

0.899 

0.944 

0.913 

0.764 

0.905 

0.875 

3 

-1 

0 

- 0.5 

- 0.029 

- 0.021 

- 0.031 

2 

1 

1.032 

0.934 

1.01 

0.604 

1.082 

0.873 

1.065 

1.098 

0.67 

1.043 

1.051 

0.918 

0.776 

0.926 

1.016 

1 

3 

0 

- 0.9 

- 0.088 

- 0.008 

- 0.027 

2 

-1 

0.813 

0.78 

0.647 

1.094 

1.055 

0.989 

1.016 

1.094 

0.688 

1.063 

0.934 

0.905 

0.705 

0.916 

0.858 

2 

-2 

2 

-1 

- 0.079 

- 0.007 

- 0.034 

2 

-1 

0.901 

1.094 

1.059 

0.631 

0.782 

0.61 

0.766 

0.711 

1.086 

0.905 

0.866 

1.045 

0.721 

1.053 

0.901 

2 

2 

3 

- 1.4 

- 0.047 

- 0.047 

- 0.042 

2 

1 

0.655 

0.83 

0.629 

1.03 

0.762 

0.764 

0.856 

0.649 

1.1 

0.799 

0.918 

0.983 

0.7 

0.879 

0.914 

2 

-3 

0 

2.2 

- 0.033 

- 0.033 

- 0.003 

1 

1 

0.926 

0.893 

1.047 

0.735 

0.674 

0.985 

0.969 

0.969 

0.735 

0.805 

0.635 

0.625 

1.051 

0.952 

0.922 

2 

3 

0 

2.8 

- 0.048 

- 0.008 

- 0.043 

1 

-1 

0.84 

0.698 

0.739 

0.973 

0.756 

1.094 

1.063 

0.875 

0.762 

0.672 

0.789 

0.672 

1.071 

1.061 

0.909 

2 

-2 

2 

1.8 

- 0.099 

- 0.019 

0.012 

1 

-2 

1.038 

0.911 

0.916 

0.717 

1.073 

0.743 

0.848 

0.653 

0.959 

0.639 

0.788 

0.686 

1.1 

0.969 

0.936 

2 

3 

3 

3.5 

- 0.077 

- 0.021 

- 0.002 

1 

1 

0.625 

0.686 

0.799 

0.952 

0.977 

0.838 

0.768 

0.713 

0.967 

0.772 

0.725 

0.702 

1.063 

0.998 

1.079 

2 

-4 

1 

0.9 

0.014 

- 0.021 

- 0.047 

1 

1 

0.686 

0.948 

0.87 

0.819 

0.965 

0.641 

0.87 

0.891 

0.713 

0.618 

0.836 

1.006 

0.612 

0.803 

0.739 

3 

3 

0 

- 1.5 

0.023 

- 0.012 

0.017 

0 

0 

0.955 

0.795 

0.608 

0.979 

0.993 

0.653 

1.092 

0.916 

0.705 

0.78 

0.901 

0.893 

0.745 

0.77 

0.774 

1 

0 

3 

- 1.2 

0.024 

- 0.045 

0.01 

2 

-1 

0.629 

0.903 

0.889 

0.7 

0.682 

0.967 

0.77 

0.754 

0.909 

0.653 

0.938 

0.856 

0.604 

0.838 

0.663 

1 

1 

2 

0.1 

0.034 

- 0.013 

0.007 

1 

2 

1.004 

0.616 

0.807 

1.09 

0.774 

0.909 

0.784 

0.625 

1.039 

0.659 

0.83 

1.004 

0.799 

0.711 

0.715 

1 

-3 

1 

1.3 

0.005 

- 0.001 

0 

1 

2 

0.688 

1.069 

0.864 

0.614 

0.721 

0.635 

0.871 

1.006 

0.623 

1.059 

0.825 

0.731 

0.971 

0.649 

0.623 

2 

1 

-1 

1.9 

0.021 

- 0.04 

0.023 

2 

0 

0.952 

0.856 

0.735 

0.942 

0.614 

0.78 

0.928 

0.887 

0.772 

1.022 

0.621 

0.809 

1.094 

0.801 

0.657 

3 

0 

2 

4 

- 0.004 

- 0.009 

0.016 

1 

-1 

0.672 

0.827 

1.034 

0.616 

0.948 

1.004 

0.672 

0.78 

1.047 

1.018 

0.721 

0.799 

0.844 

0.891 

0.746 

1 

3 

2 

3.1 

0.037 

- 0.005 

- 0.028 

2 

1 

0.918 

0.784 

0.827 

0.985 

0.83 

0.971 

0.866 

0.764 

0.957 

0.897 

0.729 

0.631 

0.989 

0.756 

0.625 
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-1 

1 

- 0.3 

- 0.056 

0.002 

0.001 

1 

2 

0.873 

0.641 

0.932 

0.666 

0.885 

1.079 

0.75 

1.049 

0.684 

0.971 

0.727 

0.942 

0.664 

0.67 

1.055 

0 

-1 

0.3 

- 0.037 

0.042 

- 0.001 

1 

0 

0.711 

0.891 

0.78 

0.95 

1.088 

0.87 

0.713 

1.053 

0.641 

0.938 

0.616 

0.924 

0.663 

0.829 

1.061 

0 

1 

- 1.7 

- 0.036 

0.036 

- 0.018 

1 

0 

1.039 

0.805 

1.08 

0.707 

0.768 

0.774 

1.038 

0.827 

1.051 

0.893 

0.807 

0.985 

0.682 

0.694 

0.897 

3 

1 

0.3 

- 0.095 

0.015 

- 0.038 

1 

1 

0.803 

1.043 

0.698 

1.018 

0.801 

0.702 

0.993 

0.846 

0.996 

0.946 

0.827 

1.063 

0.616 

0.789 

0.983 

■A 

0 

1 

- 0.075 

0.005 

- 0.045 

2 

1 

0.877 

0.791 

1.006 

0.663 

0.868 

0.893 

0.696 

0.971 

0.651 

0.631 

0.948 

0.629 

0.965 

0.725 

0.883 

1 

1 

3.4 

- 0.037 

0.015 

- 0.024 

1 

-1 

0.692 

0.862 

0.616 

1.098 

0.821 

0.993 

0.688 

0.996 

0.707 

0.797 

0.959 

0.707 

1.024 

0.682 

1.02 

-2 

3 

1.5 

- 0.054 

0.049 

- 0.009 

1 

0 

0.946 

0.629 

0.952 

0.68 

1.022 

0.829 

0.914 

0.698 

1.02 

0.856 

1.049 

0.768 

1.043 

0.75 

1.082 

1 

2 

1.3 

- 0.057 

0.035 

0.002 

1 

2 

0.733 

1.08 

0.758 

0.995 

0.893 

0.694 

0.901 

0.682 

1.088 

0.67 

0.871 

0.725 

0.934 

0.721 

1.057 

-2 

0 

- 0.2 

0.007 

0.026 

- 0.049 

0 

1 

0.786 

0.75 

1.051 

0.78 

1.071 

0.741 

0.623 

0.844 

0.721 

0.84 

0.645 

1.051 

0.76 

1.057 

0.768 

2 

1 

0.5 

- 0.007 

0.032 

- 0.033 

1 

-1 

0.94 

1.02 

0.614 

0.877 

0.864 

0.631 

0.762 

1.067 

0.858 

0.789 

0.77 

1.038 

0.805 

1.096 

0.766 

-3 

2 

0.6 

0.042 

0.033 

- 0.021 

0 

-1 

0.791 

0.661 

0.897 

0.766 

0.745 

0.924 

0.936 

0.62 

0.936 

0.717 

0.606 

0.87 

0.717 

0.909 

0.727 

1 

2 

- 1.8 

- 0.006 

0.039 

0.019 

1 

1 

1.024 

1.034 

0.719 

0.905 

0.725 

0.965 

0.981 

0.678 

0.913 

0.838 

0.743 

0.965 

0.748 

0.866 

0.61 

-1 

0 

3 

- 0.024 

0.02 

- 0.016 

1 

0 

0.649 

0.696 

0.885 

0.741 

0.739 

0.84 

0.67 

1.036 

0.799 

0.93 

1.069 

0.813 

0.887 

0.961 

0.653 

4 

1 

1.5 

0.035 

0.03 

0.014 

1 

-1 

0.844 

0.969 

0.737 

1.041 

0.702 

0.846 

0.815 

1.09 

0.795 

1.038 

1.022 

0.657 

0.932 

1.004 

0.688 

-2 

1 

2.1 

- 0.012 

0.014 

- 0.037 

1 

-2 

0.836 

0.635 

1.067 

0.83 

0.86 

1.018 

1.041 

0.795 

0.954 

0.965 

0.916 

0.645 

0.913 

1.094 

0.836 

1 

3 

2.3 

- 0.018 

0.041 

- 0.04 

0 

0 

0.903 

0.879 

0.663 

0.928 

0.981 

0.901 

0.996 

0.727 

1.057 

0.934 

0.899 

0.604 

0.911 

0.889 

0.772 

-3 

0 

-2 

- 0.049 

- 0.038 

0.056 

2 

0 

0.893 

1.053 

0.791 

0.825 

0.963 

1.063 

1.034 

0.659 

0.75 

0.%7 

0.924 

0.729 

0.743 

1.036 

0.809 

3 

1 

- 1.3 

- 0.051 

- 0.034 

0.037 

0 

-1 

0.762 

0.674 

0.926 

1.032 

1.01 

0.885 

1.086 

0.817 

0.729 

0.95 

0.883 

0.778 

0.795 

0.883 

0.807 

-2 

3 

- 0.9 

- 0.068 

- 0.039 

0.036 

1 

-2 

1.049 

1.051 

0.602 

0.69 

0.715 

0.789 

0.649 

0.942 

0.932 

1.016 

1.032 

0.834 

0.715 

0.864 

0.7 

1 

3 

0.7 

- 0.088 

- 0.045 

0.095 

1 

2 

0.694 

0.6 

0.856 

0.961 

0.625 

0.725 

0.682 

0.938 

1.043 

1.02 

0.991 

0.766 

0.643 

0.955 

0.846 

-2 

-1 

2.6 

- 0.095 

- 0.019 

0.075 

1 

1 

1.018 

1.063 

0.778 

0.647 

0.717 

0.969 

0.959 

0.791 

0.692 

0.69 

0.698 

1.082 

1.014 

0.907 

0.784 

0 

-1 

1.4 

- 0.074 

- 0.045 

0.037 

2 

-2 

0.612 

0.748 

0.977 

0.981 

0.705 

0.913 

0.895 

0.719 

0.813 

0.725 

0.61 

0.909 

0.928 

0.852 

0.649 


2 

3.5 

- 0.059 

- 0.009 

0.054 

0 

0 

0.957 

0.868 

0.731 

0.709 

0.905 

0.723 

0.639 

0.897 

1.038 

0.782 

0.637 

0.903 

0.899 

0.936 

0.848 

3 

1 

3.6 

- 0.048 

- 0.018 

0.057 

1 

1 

0.752 

0.745 

0.934 

0.852 

1.014 

0.625 

0.78 

0.991 

0.955 

0.705 

0.746 

0.967 

1.028 

1.098 

0.711 

-2 

-1 

0 

0.049 

- 0.048 

0.057 

1 

1 

0.633 

0.918 

0.77 

0.821 

1.094 

0.825 

0.875 

0.739 

0.776 

0.633 

1.043 

0.803 

0.694 

0.799 

0.895 

1 

0 

- 0.1 

0.033 

- 0.005 

0.043 

0 

-2 

1.09 

0.643 

0.95 

1.012 

0.907 

0.627 

0.881 

0.641 

0.87 

0.649 

0.887 

0.842 

0.834 

0.797 

0.957 

-4 

2 

- 0.7 

0.021 

- 0.01 

0.092 

0 

-1 

0.815 

0.887 

0.797 

0.768 

0.797 

1.084 

0.7 

0.877 

0.877 

0.846 

0.793 

0.641 

0.723 

0.612 

0.911 

3 

2 

- 1.5 

- 0.019 

- 0.012 

0.044 

1 

1 

0.973 

0.819 

1.063 

0.844 

0.764 

1.028 

0.61 

0.985 

1.014 

0.629 

1 

0.875 

0.719 

0.772 

1.008 

-2 

0 

3.8 

0.016 

- 0.016 

0.026 

1 

0 

0.825 

0.842 

0.639 

0.713 

0.791 

0.657 

1.088 

0.799 

0.76 

0.932 

0.791 

0.883 

0.948 

0.674 

1.094 

3 

0 

2.9 

- 0.014 

- 0.001 

0.085 

1 

-1 

0.996 

0.717 

0.846 

1.006 

0.698 

0.668 

0.873 

0.838 

0.832 

1.004 

0.862 

1.09 

0.858 

0.608 

0.873 

-1 

2 

2.4 

0.037 

- 0.006 

0.038 

0 

-2 

0.653 

1 

0.834 

0.745 

1.059 

0.899 

0.84 

1.012 

0.985 

1.057 

0.711 

0.852 

0.903 

0.709 

0.955 

1 

3 

1.9 

0.04 

- 0.036 

0.032 

1 

1 

1.079 

0.768 

0.881 

0.858 

1.061 

0.981 

0.737 

0.852 

1.034 

1.055 

0.608 

0.996 

1.047 

0.684 

0.95 
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2 

-1 

0 

- 1.9 

- 0.081 

0.016 

0.029 

1 

0 

0.985 

0.788 

0.645 

0.737 

0.889 

1.086 

0.629 

0.811 

0.758 

0.991 

0.805 

0.688 

0.635 

0.621 

0.631 

1 

3 

1 

0.7 

- 0.077 

0.018 

0.088 

1 

0 

0.639 

0.899 

0.868 

0.909 

0.942 

1.039 

0.778 

0.674 

0.604 

0.873 

0.758 

0.602 

0.791 

0.651 

0.813 

1 

-1 

1 

0.4 

- 0.084 

0.029 

0.079 

2 

-1 

1.041 

0.608 

0.776 

0.723 

0.817 

0.709 

0.913 

1.092 

0.944 

1.098 

0.702 

0.772 

0.688 

0.846 

0.752 

1 

0 

2 

0.3 

- 0.092 

0.042 

0.047 

1 

1 

0.618 

0.959 

1.049 

1.079 

0.649 

0.62 

0.926 

0.932 

0.854 

0.866 

0.618 

0.78 

0.608 

0.715 

0.76 

1 

-1 

-1 

2 

- 0.053 

0.046 

0.033 

2 

1 

1.059 

0.672 

0.786 

0.686 

0.62 

0.995 

0.807 

0.705 

0.621 

0.727 

1.1 

0.93 

0.975 

0.827 

0.659 

1 

1 

1 

1.1 

- 0.093 

0.032 

0.048 

1 

-2 

0.735 

1.067 

0.877 

1.067 

0.891 

0.897 

0.756 

0.672 

0.801 

0.616 

0.965 

1 

1.073 

0.645 

0.674 

1 

-4 

1 

1.4 

- 0.038 

0.023 

0.099 

0 

0 

0.92 

0.725 

0.676 

0.635 

0.846 

0.664 

1.032 

0.926 

1.026 

0.774 

0.936 

1.057 

0.996 

0.672 

0.721 

3 

0 

1 

3.1 

- 0.052 

0.016 

0.077 

2 

0 

0.805 

0.883 

1.096 

0.862 

0.989 

0.754 

1.084 

0.911 

1.002 

0.625 

0.996 

0.84 

0.918 

0.641 

0.704 

1 

-1 

0 

- 1.3 

0.047 

0.05 

0.064 

0 

2 

0.784 

0.678 

0.729 

0.661 

1.02 

0.848 

0.76 

0.612 

0.766 

0.743 

0.661 

0.612 

0.774 

0.918 

1.01 

1 

1 

-1 

- 1.1 

- 0.017 

0.039 

0.073 

2 

0 

1.034 

1.098 

0.991 

0.948 

1.084 

0.758 

0.678 

0.801 

0.803 

0.809 

0.692 

0.741 

0.661 

0.981 

1.088 

3 

-2 

1 

- 1.3 

- 0.01 

0.031 

0.074 

0 

0 

0.758 

0.84 

0.668 

0.797 

0.647 

0.938 

0.989 

1.01 

1.084 

0.885 

0.768 

0.664 

0.709 

0.977 

0.952 

1 

4 

2 

- 1.7 

- 0.007 

0.004 

0.084 

1 

0 

0.866 

1.008 

1.03 

0.883 

0.663 

1.045 

1.082 

1.03 

0.916 

0.711 

0.754 

0.739 

0.848 

1.084 

0.871 

3 

-3 

-1 

2.1 

0 

0.027 

0.055 

0 

2 

0.829 

0.829 

0.7 

0.711 

0.746 

0.7 

0.647 

0.618 

0.778 

0.987 

0.905 

1.034 

0.885 

1.086 

0.946 

2 

4 

0 

3.7 

- 0.023 

0.007 

0.071 

1 

-2 

0.934 

0.963 

0.944 

0.913 

0.827 

0.842 

0.653 

0.815 

0.62 

0.877 

1.086 

1.002 

0.875 

0.973 

1.059 

2 

-3 

3 

1.5 

- 0.016 

0.041 

0.046 

2 

-2 

0.657 

0.604 

0.659 

0.807 

1.065 

0.895 

0.905 

0.993 

1.004 

0.944 

0.92 

0.989 

1.077 

1.022 

1.098 

2 

3 

3 

2.1 

0.028 

0.043 

0.087 

0 

2 

0.879 

0.993 

0.957 

1.057 

0.913 

0.954 

0.975 

1.02 

0.852 

1.008 

1.098 

0.948 

0.969 

0.938 

0.823 

0 

3 

1 

1.7 

- 0.006 

0.04 

0.1 

2 

-1 

0.719 

0.793 

0.631 

0.692 

0.77 

0.797 

0.823 

0.629 

0.618 

0.686 

0.739 

0.721 

0.746 

0.788 

0.696 

-1 

-1 

2 

3.7 

0.008 

0.003 

0.096 

1 

0 

0.913 

0.971 

1.1 

1.043 

0.856 

0.612 

0.631 

0.668 

0.602 

0.776 

0.889 

0.719 

0.807 

0.86 

0.725 

-1 

1 

-1 

1.4 

0.044 

0.031 

0.091 

2 

1 

0.702 

0.723 

0.664 

0.672 

1.069 

1.016 

0.883 

1.065 

1.022 

0.721 

0.641 

0.676 

0.618 

0.754 

0.608 

0 

-3 

1 

1.8 

0.01 

0.025 

0.082 

2 

-1 

1.069 

0.991 

1.082 

1.077 

0.922 

0.891 

0.995 

1.1 

0.989 

0.608 

0.67 

0.705 

0.823 

0.717 

0.797 

0 

0 

1 

0.4 

0.039 

0.029 

0.063 

1 

-2 

0.643 

0.627 

0.76 

0.639 

0.852 

0.68 

0.674 

0.657 

0.739 

1.065 

0.928 

1.022 

1.03 

0.746 

0.77 

0 

-1 

3 

-1 

0.029 

0.041 

0.06 

2 

1 

1.02 

0.981 

1.02 

0.954 

1.057 

0.813 

0.791 

0.821 

0.786 

1.045 

1.036 

1.053 

1.09 

0.885 

0.616 

-2 

1 

1 

- 1.3 

- 0.023 

0.002 

0.065 

0 

1 

0.602 

0.735 

0.754 

0.76 

0.672 

1.055 

1.057 

0.93 

1.09 

0.998 

0.844 

1.08 

1.008 

0.752 

0.801 

0 

-4 

0 

0.4 

0.039 

0.024 

0.055 

1 

0 

0.832 

0.866 

1.004 

0.926 

0.813 

1.053 

0.965 

1.016 

0.907 

0.94 

1.024 

1.079 

1.061 

0.713 

0.735 

-2 

0 

3 

1.8 

- 0.042 

0.029 

0.061 

2 

-1 

1.073 

0.715 

0.829 

0.815 

0.692 

0.928 

0.803 

0.748 

0.77 

0.909 

0.694 

0.635 

0.713 

1.034 

1.049 

0 

-3 

2 

2.3 

- 0.094 

0.011 

0.035 

0 

2 

0.801 

0.942 

0.979 

1.075 

0.666 

0.879 

0.821 

0.655 

0.727 

1.012 

0.848 

0.737 

0.809 

0.858 

0.987 

-1 

3 

1 

3.8 

- 0.034 

0 

0.096 

0 

2 

0.922 

0.705 

0.705 

0.801 

0.924 

0.651 

0.985 

0.86 

0.864 

1.036 

0.815 

0.606 

0.862 

1.09 

0.969 

-2 

-3 

-1 

2.4 

- 0.045 

0.017 

0.065 

0 

0 

0.819 

0.846 

1.09 

0.918 

0.959 

0.817 

0.899 

1 

1.018 

0.907 

0.731 

0.756 

0.741 

0.993 

0.971 

-1 

4 

2 

- 1.6 

- 0.039 

0.03 

0.047 

1 

-1 

0.983 

0.704 

0.621 

0.836 

1.006 

0.934 

0.729 

0.639 

0.702 

0.819 

0.952 

1.02 

0.93 

0.94 

1.039 

-2 

-2 

2 

- 1.2 

- 0.076 

0.03 

0.098 

0 

1 

0.776 

0.989 

0.936 

0.93 

0.901 

1.034 

0.811 

0.772 

0.754 

0.821 

0.858 

1.084 

0.907 

1.032 

1.073 

-1 

1 

-1 

0.9 

- 0.082 

0.003 

0.064 

0 

1 

0.883 

0.733 

0.623 

0.674 

0.655 

0.752 

1.02 

1.079 

0.871 

0.661 

1.02 

0.871 

0.883 

0.825 

1.053 

0 

-2 

0 

- 1.6 

- 0.04 

0.022 

0.091 

1 

-2 

0.606 

0.864 

0.987 

0.889 

0.688 

0.784 

0.954 

0.965 

0.879 

0.836 

0.993 

0.926 

1.055 

1.012 

1.1 
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-1 

0 

2 

2.7 

0.035 

- 0.013 

0.046 

1 

-1 

0.854 

0.926 

0.661 

0.764 

0.766 

0.768 

1.014 

0.676 

0.655 

0.758 

1.028 

0.661 

0.803 

0.92 

0.879 

0 

-2 

2 

1.7 

- 0.02 

- 0.022 

0.078 

1 

1 

1.053 

0.721 

0.905 

0.914 

0.838 

0.686 

1.043 

0.692 

0.676 

0.842 

0.897 

0.75 

0.756 

1.08 

0.756 

-1 

2 

1 

1 

0.022 

- 0.046 

0.081 

1 

1 

0.721 

0.961 

0.678 

0.649 

0.946 

0.864 

0.793 

0.983 

0.969 

0.606 

1.073 

0.651 

0.69 

1.069 

0.707 

-1 

-2 

0 

2 

0.012 

- 0.023 

0.067 

2 

0 

1.002 

0.61 

1.028 

1.004 

0.987 

1.024 

0.743 

0.955 

0.883 

0.694 

1.08 

0.653 

0.614 

0.932 

0.664 

0 

0 

2 

- 0.8 

0.019 

- 0.043 

0.098 

2 

-1 

0.614 

0.998 

0.655 

0.684 

1.063 

0.748 

1.059 

0.645 

0.805 

0.993 

0.674 

0.946 

1.067 

0.914 

0.782 

0 

-1 

1 

- 0.8 

0.036 

- 0.036 

0.08 

1 

1 

0.971 

0.621 

1.075 

1.092 

0.842 

0.737 

0.809 

0.721 

0.723 

0.963 

0.639 

1.092 

0.973 

0.893 

0.788 

0 

2 

1 

0.8 

- 0.022 

- 0.002 

0.067 

1 

1 

0.709 

1.047 

0.686 

0.778 

0.748 

1.092 

0.651 

1.039 

1.028 

0.829 

0.737 

1.008 

0.963 

1.063 

0.741 

-1 

-1 

0 

- 0.3 

0.011 

- 0.043 

0.072 

1 

0 

0.842 

0.852 

0.967 

0.971 

0.727 

0.94 

0.6 

1.086 

1.069 

1.002 

0.774 

0.938 

0.922 

0.868 

0.83 

-1 

0 

2 

3.4 

- 0.068 

- 0.011 

0.03 

2 

-2 

0.911 

1.016 

0.746 

0.627 

0.805 

0.983 

0.983 

0.76 

0.834 

1 

0.854 

0.815 

0.659 

0.676 

1.086 

0 

-2 

3 

1.2 

- 0.041 

- 0.046 

0.053 

2 

1 

0.707 

0.727 

0.842 

1.08 

0.75 

0.856 

0.858 

0.797 

0.664 

0.817 

1.038 

0.832 

0.62 

0.87 

0.977 

-2 

2 

1 

2.9 

- 0.026 

- 0.006 

0.041 

1 

1 

0.936 

0.875 

0.745 

0.829 

0.877 

0.811 

0.664 

0.977 

1.061 

0.954 

1.077 

0.743 

0.827 

0.69 

1.028 

-1 

-3 

0 

2.5 

- 0.05 

- 0.013 

0.054 

0 

0 

0.862 

0.811 

0.883 

1.002 

1.09 

0.721 

0.752 

0.881 

1.055 

0.996 

0.891 

0.674 

0.684 

0.625 

1.067 

-1 

3 

3 

- 0.8 

- 0.07 

- 0.009 

0.075 

1 

0 

0.975 

0.987 

0.715 

0.789 

1.024 

1.002 

0.868 

0.864 

0.637 

0.731 

0.823 

0.954 

0.938 

0.733 

0.981 

-1 

-2 

1 

- 1.4 

- 0.07 

- 0.034 

0.063 

0 

2 

0.678 

0.815 

0.998 

0.969 

0.92 

0.793 

1.045 

0.766 

0.625 

0.832 

0.86 

0.916 

0.967 

0.623 

1.006 

0 

3 

0 

0.9 

- 0.029 

- 0.038 

0.051 

2 

1 

0.944 

0.897 

0.805 

0.629 

0.661 

0.692 

0.676 

0.975 

0.918 

0.778 

0.713 

0.94 

1.059 

0.704 

1.018 

-1 

0 

0 

- 0.5 

- 0.058 

- 0.01 

0.031 

2 

-1 

0.811 

0.67 

0.918 

1.049 

0.709 

0.633 

0.813 

1.057 

1 

0.668 

0.797 

0.881 

1.069 

0.758 

0.92 

0 

1 

2 

1.2 

- 0.004 

0.05 

- 0.035 

1 

-1 

0.67 

0.614 

0.891 

0.813 

0.67 

0.69 

0.694 

0.95 

0.608 

0.676 

0.705 

0.955 

0.702 

0.657 

1.034 

0 

0 

2 

2.1 

0.028 

0.047 

0.02 

0 

2 

0.907 

1.082 

0.682 

0.975 

0.789 

0.713 

0.627 

1.077 

0.815 

0.623 

0.715 

0.846 

0.819 

0.737 

0.844 

-1 

3 

0 

1.6 

0.031 

0.044 

- 0.022 

0 

0 

0.705 

0.688 

1.008 

0.645 

0.875 

1.022 

1.096 

0.778 

0.873 

0.723 

0.733 

1.016 

0.784 

0.629 

1.024 

-1 

-2 

1 

2.2 

0.017 

0.038 

- 0.012 

2 

0 

1.026 

1.075 

0.707 

0.895 

0.871 

0.973 

1.094 

0.737 

0.983 

0.621 

0.633 

0.952 

0.754 

0.813 

1.03 

0 

1 

1 

-1 

0.048 

0.025 

- 0.019 

1 

0 

0.623 

0.754 

1.073 

0.827 

0.834 

0.629 

0.698 

0.959 

0.719 

0.961 

1.047 

0.668 

1.032 

0.692 

0.942 

0 

-2 

2 

- 0.5 

0.03 

0.023 

- 0.036 

0 

0 

1.084 

1.024 

0.717 

1.022 

0.881 

0.674 

0.721 

0.957 

0.819 

0.981 

1.057 

0.67 

0.95 

0.635 

0.838 

0 

0 

1 

0 

0.017 

0.027 

0.022 

1 

1 

0.664 

0.756 

0.996 

0.762 

0.696 

1.03 

0.942 

0.616 

0.862 

1.08 

1.088 

0.801 

0.836 

0.805 

0.889 

0 

-1 

0 

0.8 

0.022 

0.02 

- 0.011 

1 

0 

0.977 

1.032 

0.848 

0.967 

0.657 

1.082 

0.991 

0.756 

1.053 

0.936 

1.045 

0.623 

0.993 

0.7 

0.998 

-1 

2 

3 

1.8 

- 0.03 

0.024 

- 0.018 

1 

-1 

0.95 

0.776 

0.913 

0.776 

0.653 

0.918 

0.782 

0.954 

0.797 

0.848 

0.604 

0.977 

0.86 

1.002 

0.819 

-1 

-3 

2 

3.1 

- 0.053 

0.007 

0.009 

0 

2 

0.848 

0.922 

0.711 

0.944 

0.684 

0.955 

0.62 

0.871 

0.606 

0.875 

0.684 

0.879 

0.821 

1.041 

0.84 

-1 

4 

0 

3.2 

- 0.041 

0.048 

0.024 

2 

2 

0.928 

0.694 

0.948 

0.809 

1 

0.756 

0.967 

0.832 

0.875 

0.985 

0.686 

0.973 

0.621 

1.045 

0.637 

-2 

-3 

0 

2.7 

- 0.054 

0.027 

0 

1 

-1 

0.809 

1.088 

0.862 

1.088 

0.996 

0.868 

0.946 

0.776 

0.911 

1.053 

0.69 

0.936 

0.832 

0.959 

0.686 

-1 

3 

2 

- 1.9 

- 0.075 

0.014 

- 0.02 

0 

-1 

0.954 

0.623 

1.016 

0.618 

0.957 

0.834 

0.602 

0.948 

0.84 

0.752 

1.041 

0.786 

0.889 

1.039 

0.733 

-1 

-2 

2 

- 1.2 

- 0.063 

0.049 

- 0.02 

2 

1 

0.635 

0.936 

0.725 

0.84 

1.049 

0.905 

0.776 

0.914 

0.811 

0.786 

1.053 

0.827 

1.075 

1.1 

0.698 

-1 

1 

0 

- 0.7 

- 0.051 

0.026 

0.021 

0 

1 

0.83 

0.663 

0.973 

0.834 

0.623 

0.704 

1.067 

0.696 

1.065 

0.813 

0.955 

0.789 

0.871 

1.038 

0.795 

0 

-3 

0 

- 0.9 

- 0.031 

0.044 

- 0.008 

0 

-1 

0.731 

1.041 

0.86 

0.916 

0.621 

0.649 

0.911 

0.733 

0.856 

0.87 

0.914 

0.633 

0.942 

1.073 

0.678 
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-2 

2 

2 

2.4 

0.05 

- 0.014 

0.006 

0 

-2 

0.637 

1.036 

0.959 

0.653 

0.733 

0.639 

1.039 

0.895 

0.78 

0.748 

0.981 

1.01 

0.655 

0.881 

0.963 

-1 

-4 

3 

1.4 

0.046 

- 0.035 

- 0.008 

2 

0 

1.1 

0.743 

0.772 

0.846 

0.604 

0.623 

1.008 

1.096 

0.725 

0.674 

0.977 

0.907 

0.68 

1.014 

1.071 

-2 

1 

0 

3.9 

0.041 

- 0.037 

- 0.044 

1 

2 

0.684 

1.01 

0.938 

0.799 

0.914 

1.057 

0.739 

0.702 

1.067 

0.666 

1.018 

1.041 

0.739 

1.047 

0.995 

0 

-4 

-1 

2.6 

0.032 

- 0.028 

- 0.01 

1 

-1 

1.092 

0.682 

0.606 

1.045 

0.916 

1.1 

0.748 

0.807 

0.893 

0.788 

1.012 

1.1 

0.846 

1.02 

0.885 

-1 

4 

2 

0.1 

0.013 

- 0.048 

- 0.039 

0 

0 

0.741 

1.061 

1.026 

0.754 

1.092 

0.688 

0.998 

1.038 

0.709 

0.955 

0.717 

0.713 

0.92 

0.965 

0.907 

-2 

-2 

1 

- 1.9 

0.01 

- 0.037 

- 0.029 

1 

1 

0.961 

0.655 

0.612 

0.868 

0.94 

0.663 

1.01 

0.858 

0.694 

1.086 

0.629 

0.639 

1.049 

0.922 

0.868 
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APPENDIX F. MSM CLASS ORGANIZATION 


Source Package 


W data.hist ffi model.structure 


1^ GainDate.java 

Bl GAR.java 


LossData.java 

•• Bl GarTre.java 


PromotionData.java 

Bl Marine.java 

W data.user 

Bl Mos.java 


Data.java 

• Bl Network.java 


^ DataByFy.java 

Bl Node.java 


1^ DataByFyGrade.java 

••-B| SNode.java 


§ Dat^yFyMos.java B oracle 


•-B| DataSet.java 

Bl OradeData.java 


Bl DateFromSeq.java 

Bl OradeDateFromSeq.java 


Bl Distributions .java 

Bl OradeGains.java 


ForceControls.java 

Bl OradelnvSeq.java 


InvSeq.java 

- Bl OradeLosses.java 


Inventory .java 

Bl OradeODSEInventory.java 


^ PayData.java 

Bl OradePromotions.java 


^ RunArgs.java 

Bl OradeSequence.java 


Sequence .java 

Bl OradeTFDWInventory.java 


Igmain 

- Bl OradeTargetDistros.java 


InputPrep.java B 


InventoryDataTest.java 

Bl ArmualStatsAggregator.java 


fe Main.java 

Bl MonthlyS tats Aggregator .java 


Bl Simulation.java 

Bl l^esultsManager.java 


^ main.gui 

Bl Writer.java 


|B| GenericAction.java B 


Bl MSMMenubar.java 

-B^ TestCreateInputs.java 


Bl MSMToolBar.java 

B^ TestPOI.java 


•• ManGui.java W.jUtils 


B) WindowCloser.java 

Bl Exce^leType.java 

W model.process 

Bl TabNotFoundException.java 


Bl Accessions.java 

Bl UnsupportedRIeTypeException.java 


Bl Oassification.java 

-B| Utils.java 


Bl LatMove.java 
-Bl LatMovePromotions.java 
•• Bl^osses.java 

Bl Process.java 

Bl Promote.java 

Bl RegularPromobons.java 
- Bl Retention .java 

Bl Version .java 


Test Package 

ffi utils 

- [jj^xceTileTypeTest.java 
i.- - l^flVersionTest.iava 


Libraries 

Libraries 

® S Apache POI-poi-3.ll-20141221.jar 

(i) B Apache POI - poi-ooxml-3. ll-20141221.jar 

0 B Apache POI-poi-ooxml-schemas-3.ll-20141221.jar 

S) B Apache POI - xmlbeans-2.6.0.jar 

(jl B Apache Commons Math - commons-math3-3.5.jar 

0 B MersenneTwisterFast - MersenneTwisterFast.jar 

0 B ojdbc-ojdbcH.jar 

ij) B - junit-4.10.jar 

i s JDK 1.8 (Default) 


Figure 53. MSM Class Organization. Adapted from Garrick (2014). 
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APPENDIX G. VALIDATION RESULT TABLES 


Table 11. NEAS Losses Relative Error 



lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

-54% 

-49% 

-45% 

-43% 

-47% 

2005 

-53% 

-47% 

-43% 

-39% 

-47% 

2006 

-59% 

-56% 

-51% 

-51% 

-51% 

2007 

-52% 

-49% 

-48% 

-41% 

-44% 

2008 

-53% 

-54% 

-46% 

-42% 

-47% 

2009 

-51% 

-46% 

-40% 

-40% 

-46% 

2010 

-43% 

-35% 

-32% 

-35% 

-44% 

2011 

-38% 

-35% 

-34% 

-35% 

-49% 

2012 

-33% 

-33% 

-31% 

-43% 


2013 

-33% 

-32% 

-41% 



2014 

-33% 

-44% 





Table 12. Retired Losses Relative Error 



lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

-25% 

-39% 

-23% 

-43% 

-20% 

2005 

-18% 

-29% 

-13% 

-39% 

-32% 


-17% 

-22% 


-49% 


wamrM 


-22% 

-26% 



2008 

-22% 

-50% 

-25% 

-41% 

-49% 

2009 

-16% 

-11% 

-11% 

-53% 

-31% 

2010 

18% 

-8% 

-26% 

-29% 

-58% 

2011 

-6% 

-44% 

-20% 

-60% 

-79% 

2012 

-22% 

-26% 

-47% 

-81% 


2013 

-5% 

-48% 

-71% 



2014 

-33% 

-72% 





Table 13. Other Losses Relative Error 



lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

-29% 

-11% 

-19% 

-12% 

16% 

2005 

-8% 

-20% 

1% 

47% 

20% 

2006 

-40% 

-23% 

28% 

8% 

20% 

2007 

-6% 

74% 

82% 

112% 

123% 

2008 

-22% 

18% 

49% 

97% 

73% 

2009 

-21% 

-6% 

27% 

25% 

68% 

2010 

3% 

36% 

62% 

116% 

62% 

2011 

7% 

11% 

74% 

43% 

5% 

2012 

-5% 

31% 

23% 

-9% 


2013 

6% 

-3% 

-21% 



2014 

6% 

-23% 
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Table 14. EAS Losses Relative Error 



lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

-6% 

-3% 

7% 

11% 

-49% 

2005 

-4% 

-2% 

14% 

29% 

-53% 

2006 

-7% 

4% 

32% 

24% 

-55% 

2007 

-2% 

19% 

23% 

11% 

-58% 

2008 

-8% 

-6% 

-5% 

-5% 

-57% 

2009 

-15% 

-19% 

-9% 

-10% 

-50% 

2010 

-16% 

-12% 

-4% 

-6% 

-56% 

2011 

-10% 

-10% 

-1% 

-17% 

-28% 

2012 

-4% 

-3% 

-10% 

1% 


2013 

-6% 

-18% 

2% 



2014 

-22% 

-12% 





Table 15. Aggregated Gains Relative Error 



lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

2% 

-41% 

-34% 

-42% 

-38% 

2005 

-8% 

-40% 

-46% 

-38% 

-26% 

2006 

1% 

-52% 

-38% 

-19% 

49% 

mm 

-28% 

-52% 


46% 

63% 

mm 

13% 



120% 

243% 

2009 

16% 

34% 

77% 

197% 

253% 

2010 

66% 

41% 

153% 

209% 

237% 

2011 

14% 

39% 

87% 

107% 

43% 

2012 

46% 

59% 


29% 


2013 

16% 

29% 




2014 

19% 

-23% 





Table 16. Accession Gains Relative Error 



lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

-100% 

-28% 

14% 

-4% 

-29% 

2005 

-100% 

-5% 

-1% 

10% 

-7% 

2006 

-100% 

-20% 

8% 

36% 

-32% 

2007 

-100% 

0% 

37% 

18% 

-10% 

2008 

-100% 

4% 

2% 

29% 

-16% 

2009 

-100% 

-17% 

27% 

24% 

-64% 

2010 

-100% 

26% 

35% 

-24% 

-49% 

2011 

-50% 

30% 

-18% 

-5% 

-33% 

2012 

-5% 

-21% 

3% 

-9% 


2013 

-19% 

-6% 

-7% 



2014 

-24% 

-19% 
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Table 17. Promotions to E-2 Relative Error 



lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

-40% 

-68% 

-20% 

-5% 

-30% 

2005 

-44% 

-60% 

-11% 

-12% 

-19% 

2006 

-42% 

-62% 

-17% 

-1% 

-2% 

2007 

-43% 

-55% 

-4% 

21% 

-2% 

2008 

-44% 

-63% 

-3% 

9% 

-1% 

2009 

-41% 

-64% 

1% 

19% 

-26% 

2010 

-44% 

-41% 

21% 

-4% 

-37% 

2011 

-21% 

-15% 

-2% 

-14% 

-18% 

2012 

3% 

-18% 

-10% 

1% 


2013 

-5% 

-11% 

-1% 



2014 

-6% 

-16% 





Table 18. Promotions to E-3 Relative Error 



lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

-11% 

-69% 

-42% 

-4% 

-13% 

2005 

-10% 

-69% 

-28% 

-8% 

-12% 

2006 

-10% 

-67% 

-32% 

-8% 

1% 

mm 


-67% 


5% 

12% 

mm 




-1% 

2% 

2009 

-10% 

-68% 

-31% 

5% 

3% 

2010 

-13% 

-64% 

-10% 

12% 

-29% 

2011 

-5% 

-37% 

-2% 

-21% 

-19% 

2012 

-1% 

-10% 

-26% 

-9% 


2013 

-1% 

-21% 

-12% 



2014 

-4% 

-20% 





Table 19. Promotions to E-4 Relative Error 



lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

-61% 

-46% 

-18% 

-27% 

-20% 

2005 

-67% 

-20% 

-24% 

21% 

-38% 

2006 

-62% 

-33% 

16% 

7% 

-50% 

2007 

-66% 

9% 

4% 

-6% 

-41% 

2008 

-67% 

-25% 

-36% 

-24% 

-73% 

2009 

-60% 

-40% 

-28% 

-48% 

-55% 

2010 

-63% 

-24% 

-42% 

-34% 

-53% 

2011 

-40% 

-35% 

-24% 

-29% 

-41% 

2012 

-41% 

-16% 

-19% 

-18% 


2013 

-23% 

-20% 

-11% 



2014 

-25% 

-22% 
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Table 20. Promotions to E-5 Relative Error 



lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

-94% 

-65% 

-42% 

-51% 

-15% 

2005 

-93% 

-51% 

-44% 

-7% 

-9% 

2006 

-88% 

-58% 

-6% 

-3% 

-23% 

2007 

-89% 

-20% 

1% 

-9% 

-5% 

2008 

-82% 

-39% 

-49% 

-24% 

-68% 

2009 

-68% 

-50% 

-16% 

-59% 

-44% 

2010 

-84% 

-27% 

-57% 

-42% 

-70% 

2011 

-49% 

-50% 

-20% 

-51% 

-59% 

2012 

-69% 

-25% 

-42% 

-49% 


2013 

-31% 

-37% 

-30% 



2014 

-31% 

-39% 





Table 21. Promotions to E-6 Relative Error 



lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

-96% 

-54% 

-43% 

-36% 

-2% 

2005 

-93% 

-56% 

-18% 

-3% 

-5% 

2006 

-77% 

-32% 

10% 

-4% 

-40% 

mm 

-73% 

-4% 

6% 

-42% 


mm 

-44% 

-11% 


-8% 

-49% 

2009 

-29% 

-43% 

3% 

-52% 

-9% 

2010 

-44% 

4% 

-32% 

1% 

-74% 

2011 

-29% 

-41% 

17% 

-65% 

-47% 

2012 

-43% 

14% 

-48% 



2013 

19% 

-50% 

-21% 



2014 

-8% 

-19% 





Table 22. Promotions to E-7 Relative Error 



lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

-91% 

-66% 

-43% 

-37% 

21% 

2005 

-88% 

-65% 

-11% 

12% 

-19% 

2006 

-79% 

-39% 

31% 

-30% 

-39% 

2007 

-78% 

7% 

-13% 

-48% 

11% 

2008 

-50% 

-31% 

-42% 

-1% 

-41% 

2009 

-40% 

-30% 

24% 

-43% 

-16% 

2010 

-40% 

30% 

-10% 

-6% 

-55% 

2011 

-24% 

-34% 

0% 

-59% 

-42% 

2012 

-31% 

-6% 

-30% 

-42% 


2013 

11% 

-35% 

-17% 



2014 

-23% 

-30% 
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Table 23. Promotions to E-8 Relative Error 



lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

-94% 

-96% 

-40% 

-57% 

67% 

2005 

-96% 

-69% 

-35% 

45% 

6% 

2006 

-90% 

-61% 

68% 

-16% 

-27% 

2007 

-90% 

36% 

22% 

-44% 

20% 

2008 

-42% 

-16% 

-32% 

-4% 

-32% 

2009 

-25% 

-10% 

42% 

-29% 

-6% 

2010 

-9% 

48% 

8% 

0% 

-20% 

2011 

-5% 

-26% 

4% 

-27% 

-30% 

2012 

-15% 

-2% 

9% 

-37% 


2013 

16% 

4% 

5% 



2014 

-26% 

-23% 





Table 24. Promotions to E-9 Relative Error 



lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

-93% 

-97% 

-42% 

-47% 

125% 

2005 

-96% 

-74% 

-29% 

101% 

11% 

2006 

-92% 

-72% 

134% 

-21% 

13% 

mm 


74% 

16% 

-21% 

33% 

mm 

-18% 

-29% 

-1% 

-9% 

-10% 

2009 

-6% 

53% 

45% 

-6% 

21% 

2010 

24% 

38% 

27% 

3% 

1% 

2011 

7% 

-15% 

18% 

-26% 

-5% 

2012 

5% 

37% 

23% 

-22% 


2013 

62% 

5% 

15% 



2014 

-19% 

-6% 





Table 25. End Strength Relative Error 



lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

-15% 

-20% 

-18% 

-19% 

-19% 

2005 

-17% 

-18% 

-19% 

-19% 

-15% 

2006 

-16% 

-19% 

-19% 

-15% 

-12% 

2007 

-19% 

-19% 

-15% 

-12% 

-6% 

2008 

-17% 

-15% 

-12% 

-6% 

0% 

2009 

-13% 

-12% 

-6% 

0% 

-2% 

2010 

-12% 

-6% 

0% 

-2% 

0% 

2011 

-6% 

0% 

-2% 

0% 

0% 

2012 

0% 

-2% 

0% 



2013 

-2% 

0% 

0% 



2014 

0% 

0% 





127 







Table 26. Retention Zone-A Relative Error 



lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

3% 

-6% 

-6% 

-49% 

-90% 

2005 

-1% 

-4% 

-51% 

-40% 

-84% 

2006 

8% 

-48% 

-39% 

-28% 

-80% 

2007 

-42% 

-36% 

-32% 

-23% 

-68% 

2008 

4% 

-3% 

3% 

38% 

-43% 

2009 

11% 

11% 

84% 

28% 

-53% 

2010 

-2% 

53% 

24% 

-7% 

-41% 

2011 

55% 

19% 

-1% 

11% 

11% 

2012 

-2% 

-18% 

-2% 

8% 


2013 

-10% 

6% 

15% 



2014 

21% 

33% 





Table 27. Retention Zone-B Relative Error 



lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

5% 

4% 

31% 

-3% 

10% 

2005 

-4% 

-10% 

-8% 

13% 

11% 

2006 

-8% 

-29% 

23% 

38% 

57% 

mm 

-29% 

-9% 

46% 

83% 


mm 

9% 

34% 

117% 

100% 

97% 


18% 

91% 

121% 

137% 

168% 

2010 

59% 

77% 

131% 

160% 

128% 

2011 

37% 

74% 

136% 

117% 

80% 

2012 

41% 

94% 

113% 

78% 


2013 

55% 

69% 

83% 



2014 

44% 

51% 





Table 28. Retention Zone-C Relative Error 



lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

19% 

34% 

43% 

-3% 

9% 

2005 

22% 

27% 

1% 

19% 

31% 

2006 

17% 

-4% 

35% 

58% 

50% 

2007 

-12% 

20% 

69% 

69% 

22% 

2008 

16% 

63% 

95% 

64% 

35% 

2009 

28% 

72% 

60% 

41% 

55% 

2010 

36% 

44% 

42% 

67% 

110% 

2011 

10% 

16% 

57% 

86% 

64% 

2012 

-5% 

48% 

114% 

86% 


2013 

36% 

123% 

97% 



2014 

92% 

89% 
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Table 29. Retention Zone-D Relative Error 



lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

4% 

18% 

42% 

-11% 

29% 

2005 

8% 

33% 

-4% 

33% 

77% 

2006 

18% 

-7% 

51% 

98% 

18% 

2007 

-17% 

45% 

116% 

24% 

34% 

2008 

31% 

107% 

43% 

51% 

31% 

2009 

65% 

38% 

55% 

34% 

18% 

2010 

12% 

48% 

37% 

25% 

75% 

2011 

24% 

28% 

38% 

75% 

75% 

2012 

8% 

29% 

96% 

93% 


2013 

12% 

85% 

123% 



2014 

51% 

101% 





Table 30. Retention Zone-E Relative Error 



lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

153% 

183% 

195% 

75% 

187% 

2005 

178% 

188% 

101% 

214% 

233% 

2006 

186% 

116% 

295% 

309% 

138% 

WiAtVM 




184% 

154% 

2008 


451% 




2009 

452% 

303% 

225% 

209% 

192% 

2010 

268% 

243% 

227% 

213% 

261% 

2011 

240% 

244% 

299% 

313% 

256% 

2012 

227% 

287% 

417% 

312% 


2013 

269% 

407% 

423% 



2014 

370% 

415% 





Table 31. Retention Zone-Senior Enlisted Relative Error 



lyr-Forecast 

2yr-Forecast 

3yr-Forecast 

4yr-Forecast 

5yr-Forecast 

2004 

133% 

112% 

127% 

-1% 

398% 

2005 

121% 

90% 

22% 

173% 

116% 

2006 

106% 

22% 

302% 

31% 

43% 

2007 

26% 

300% 

92% 

-4% 

83% 

2008 

392% 

161% 

89% 

98% 

101% 

2009 

82% 

52% 

67% 

12% 

53% 

2010 

45% 

58% 

36% 

15% 

44% 

2011 

114% 

91% 

108% 

60% 

87% 

2012 

86% 

105% 

108% 

46% 


2013 

76% 

101% 

79% 



2014 

128% 

124% 
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